Six words a verification badge would need to mean anything

Follow a laboratory report from the bench to a product page and watch the information fall away. The report states a method, a date, a sample identifier, a submitter and a result. The vendor extracts the result. The listing extracts the fact that a result exists. The badge extracts the word verified. At each step the claim gets shorter, more general and more durable, until what began as a measurement on one vial on one day has become a permanent property of a product line, asserted in a graphic that carries no date.

Who chose the vial

The identity of the submitter is the single most informative piece of context attached to any third-party report, and it is the piece most often missing when a report is quoted. A test commissioned by a buyer on a vial bought anonymously at retail is a different evidential object from a test commissioned by the vendor on a vial it selected from its own stock. Both are legitimate. They support different inferences, and the difference is not small.

In the buyer-submitted case the vial entered the postal system as an ordinary retail unit and its selection was outside the vendor’s control, which is exactly the property that makes the result informative about what a customer receives. In the vendor-submitted case the vendor chose the vial, knew it was being tested, and will decide whether the result is published. Nothing improper need occur for the second to produce a more flattering picture than the first, and no laboratory can distinguish them from the bench.

The Journal’s practice is to state the submitter type wherever it cites a figure, and to decline to cite a figure where the submitter is unknown. This is more restrictive than the market’s convention and it means we publish fewer numbers. It also means that when we say a result came from an anonymous retail purchase, that claim is doing work.

A worked estimate of the selection effect

Since the unpublished results are invisible, any estimate of the selection effect has to be constructed rather than measured, and the Journal offers the following as an illustration rather than a finding. Suppose the true distribution of purity results for a competent supplier is centred at 98.0% with a standard deviation of 0.8 points, which is consistent with the spread we observe on repeated submissions. Suppose the supplier publishes results above 98.0% and files the rest.

The published mean is then approximately 98.6%, the published minimum is 98.0%, and the apparent variability is roughly halved. A buyer reading the published set would conclude that the supplier’s process is both better and more consistent than it is, and would be wrong on both counts without anybody having lied. Increase the publication threshold to 98.5% and the published mean rises to 99.0% while the true mean is unchanged.

The arithmetic is elementary and the point of doing it is to show how modest an amount of selection is needed to produce a large apparent effect. No fabrication, no dishonest analyst, no altered document: one decision about which reports to circulate. Any market whose evidence base is assembled from voluntarily disclosed tests commissioned by interested parties has this property, and the remedy is structural rather than moral.

A laboratory has no sanction

A frequently overlooked asymmetry: none of these services has any authority over a vendor. A laboratory that finds a submitted sample at 91% purity cannot compel a recall, cannot require a retest, cannot publish the finding over the client’s objection without breaching confidentiality, and cannot prevent the vendor from continuing to advertise a figure obtained on a different lot. It can decline further business, which is a real sanction and a slow one.

This is not a shortcoming of the services. Confidentiality to the client is a requirement of the accreditation framework, not an indulgence, and a laboratory that published clients’ results unilaterally would be a worse institution, not a better one. But it means the word verification is doing something the underlying arrangement cannot support: verification in ordinary usage implies a check that can fail with consequences, and here the only consequence of a bad result is that nobody hears about it.

The one structural exception is the public archive. Where a service records that a submission occurred, a vendor cannot quietly discard an unfavourable result, because the fact of the test is on the record even if the detail is not. That is why this article treats the archive as the most important product feature in the sector, and why the Journal’s standing request to all four services is that the existence of a submission be public even where the result is confidential.

The report is the product

Across the four services the Journal has catalogued eleven elements that appear on some reports and not others: the compound and lot as declared by the submitter; the date of receipt; the condition of the sample on arrival; the instrument; the column; the gradient; the detection wavelength; the integration threshold; the reproduced chromatogram; a named analyst; and an explicit statement that results apply to the sample as received.

No service omits all of these and none includes all of them on every tier. The best reports in the sector carry nine or ten and are genuinely good documents by any standard — better, in several respects, than the manufacturer certificates they are checking. The thinnest carry three, and a three-element report is a number with a letterhead.

The reporting requirements in the accreditation standard are the natural benchmark here, and they are not demanding: identification of the items tested, the date of receipt, the methods used, the results with units, and a clear statement of what the results apply to.¹ A report meeting that list is checkable. The Journal’s standing request to all four services is a single addition beyond it — print the gradient and the integration threshold — because those two numbers are what make a purity figure comparable with anybody else’s.

The archive, and why it matters more than the method

A public, searchable archive of prior results changes the epistemics of this market more than any improvement in analytical method would. The reason is the selection problem. A single report is a snapshot chosen by whoever paid for it. A record of every submission relating to a vendor, published whether or not the result flattered the submitter, is a series — and a series cannot be curated by choosing which snapshots to show.

Two of the four services maintain something of this kind. The Journal would like all four to, and has asked. The principal objection raised in response is a real one: client confidentiality is an obligation, not a preference, and a laboratory cannot unilaterally publish a paying client’s results. We accept that entirely, which is why our request is narrower. Publish the fact of the submission — the date, the vendor named by the submitter, and the compound — while keeping the result confidential where the client requires it.

That narrower version defeats most of the selection effect at no cost to confidentiality. A vendor with nine public submissions and three published results is visibly a vendor with six results it declined to publish, and a reader can price that in. It is, so far as we can tell, the single highest-leverage change available to this sector, and it requires no new equipment and no new method.

What a badge asserts, and what it supports

A verification badge on a product listing typically asserts, in one or two words and a graphic, that the product has been tested by a named service. Consider what that claim leaves open. Which lot was tested. When. Who submitted the sample and how it was selected. What was measured — purity alone, or identity, or content. By what method. Whether the lot currently on sale is the lot that was tested. Whether the report is available to the reader.

Every one of those is material and every one is absent. A badge is therefore not evidence; it is an assertion that evidence exists, offered without the particulars that would let anybody assess it. The Journal does not treat badges as evidence in its coverage, states so wherever it reports one, and will not cite a badge as support for a claim about material.

The remedy is four data points the vendor already possesses: the lot number, the date of analysis, the service, and a link to the report. Several listings in this market already carry them, which establishes both that it is possible and that it is not commercially fatal. What it requires is that a badge become perishable — attached to a lot rather than to a product line — and perishability is precisely the property a marketing asset is designed not to have.

How a claim decays between the bench and the listing

The decay is worth tracing precisely, because at no step does anybody say anything false. The laboratory reports a purity figure for the sample as received, on a stated date, by a stated method, submitted by a named party. The vendor extracts the figure and the service name onto its own documentation, dropping the submitter and often the method. A reseller reproduces the vendor’s documentation, dropping the date. A listing summarises the whole chain as tested by a named service. A badge reduces it to verified.

Each step is a reasonable act of summarisation and the cumulative effect is a claim of a different kind from the one the laboratory made. A time-indexed measurement on one sample has become an atemporal property of a product line. The information was not concealed; it was compressed away by a chain of parties each of whom had a legitimate reason to shorten the message.

This is why the Journal reports the provenance of every third-party figure it cites — which service, which lot, which date, which submitter — and treats a figure lacking any of those as uncitable. It makes our coverage sparser than the market’s. It also means that a number appearing in this publication can be traced to a document, which is the only property that distinguishes reporting from repetition.

Four changes, none requiring a regulator

First, publication of submissions rather than only of results: the date, the vendor as named by the submitter, and the compound, with the result confidential where the client requires it. This defeats most of the selection effect and costs nothing.

Second, submitter type on every report — vendor, buyer, publication or reseller — which the laboratory knows and which determines what the result can support. Third, lot-linked badges carrying a lot number, a date and a link, so that a verification claim expires with the lot it describes. Fourth, gradient and integration threshold printed on every purity report, which is the only way two figures from different services can be compared at all.

None of the four requires a regulator, new legislation, or any change in analytical practice. Three of them require a laboratory to print information it already holds; the fourth requires a vendor to accept that a badge should perish. The Journal has put all four to each of the services covered here. Responses have been mixed and mostly constructive, and are printed in our correspondence pages as they arrive. We will publish an annual note on which have been adopted, because the alternative is making the same request indefinitely without recording the answer.

Our own position, stated in full

Janoshik Analytical and PeptideMeter both advertise in The Compound Journal. Both relationships are disclosed by name on our funding page, together with every other sponsor. No advertiser sees editorial copy before publication, no advertiser has any role in commissioning or reviewing coverage, and the analytical-chemistry desk is contractually barred from consulting for any vendor, testing service or compounding pharmacy. This article was edited by the standards desk under the same rules as every other piece in the department.

The Journal also pays these services. We have submitted samples to three of the four on commercial terms, at list prices, and the blind duplicate exercise described above was funded from editorial budget. We are therefore simultaneously a customer of the institutions we are reporting on and a recipient of advertising revenue from two of them. Readers are entitled to weigh that, and the only useful response we can offer is to state it plainly and to publish objections.

Our position on the substance is unchanged by any of it. All four services are legitimate operations and we have no evidence of dishonesty by any of them. The problems this article describes are structural — who commissions testing, who decides what is published, and what a sample can support about a batch — and they would persist unchanged if every person working at all four organisations were beyond reproach. Correspondence to standards@compoundjournal.com.

What the Journal thinks these services are worth

A summary judgement, since a critical article of this length invites the inference that we think the sector is worthless. We do not. Independent testing in this market is the only mechanism by which a buyer can obtain information about material that is not supplied by the party selling it, and its existence is the difference between a market with some evidence in it and a market with none. Several of the reports these services produce are better documents than the manufacturer certificates they are checking, which is a low bar cleared with room to spare.

The three criticisms we would press are narrow. A sample is not a batch, and the trade cites samples as batches. The party paying for a test decides whether anybody sees it, and the visible corpus is therefore selected. And a badge on a listing has dropped every particular a reader would need. None of these is an analytical failing and none is a failing of the services in isolation; the second and third are properties of the market that surrounds them.

What we would tell a reader is this. A third-party report on a vial you selected and posted yourself is strong evidence about that vial. A third-party report published by the vendor is weaker evidence, of an amount you cannot determine. A badge is not evidence. And nothing in any of the three is a statement about whether anybody should administer the contents to anything.

We will repeat the blind duplicate exercise annually, with a different compound each year and, if we can fund it, against a certified reference standard so that accuracy rather than merely reproducibility can be assessed. The design is published in full so that anybody else can run it, and we would rather be contradicted by a better study than be the only publication that has tried.