How a rating is made
PQC Arena is a relative, editorial rating built on evidence anyone can check. This page states how the evidence is gathered, what turns it into a tier, and — the part most ratings leave out — the conditions under which it refuses to produce one.
It is relative and editorial, not mechanical
A vendor is compared against its peer set on each dimension rather than scored out of a total. There is no 1–100 figure and no weighted formula, because several of the ten dimensions are irreducibly qualitative and a number would lend them a precision they do not have.
Arena’s defensibility therefore does not rest on a formula. It rests on four things a reader can check for themselves: criteria published before anyone is rated, a citation on every claim, a right of reply for any rated party, and conformance gates that override an otherwise strong result.
Vendor-neutral means no stake in who wins — not a refusal to compare
The Q-Day Index declines to name a winner because its subjects are research machines measured against a physics target, and ranking them would misrepresent what the measurement means. Arena’s subjects are commercial products that a buyer has to choose between. Declining to compare them helps nobody; it just leaves the comparison to sales collateral.
Neutrality here means Q-Advantage has no stake in which vendor wins: it does not resell, integrate, or take commission on any rated product, and no vendor can buy a rating, a tier, or its timing.
Four evidence tracks
The first three need no cooperation from the vendor at all. That is deliberate: a rating that only works for vendors who reply would really be a rating of who answers email.
- 1Public claims auditThe primary track. Read the vendor's own published documentation, case studies and whitepapers, and score what is and is not substantiated. Needs no cooperation and runs for every candidate.
- 2Independent testingWhere an implementation exposes a public SDK, library or endpoint, run it through Q-Shield directly. This is what moves correctness and performance from claim to measurement. Where an implementation is not testable, the dimension is recorded as not assessed — never inferred from the vendor's own figures.
- 3Certification-registry checksRead the public registries and record what was actually validated, at what scope and version. Certificate scope is read, not just its existence: what is validated is frequently narrower than what is marketed.
- 4Vendor cooperation, optionalVendors are invited to provide deeper access, and receive advance notice of the criteria and a pre-publication correction window. No fee is charged and none is accepted. A vendor that declines is still rated on tracks 1–3, and lands in Unavailable only if there is genuinely nothing public to audit.
A tier requires 7 of 10 dimensions
This is the number that decides whether Arena is allowed to have an opinion about a vendor at all. Below 7 assessed dimensions, a vendor gets no tier — not a low one, and the rating says why in place of a verdict.
A tier assembled from two or three dimensions would be the exact overconfident number this methodology exists to prevent. The threshold is enforced in code rather than by editorial discipline: a rating that carries a tier it is not eligible for fails the build and cannot be published.
Conformance gates
Some failures cannot be traded off against strength elsewhere. Failing known-answer tests against the standard a vendor itself claims conformance to lands that vendor in Underperform regardless of how it scores on the other nine dimensions. A gate is always assignable — a vendor whose cryptography is provably wrong does not get to be unrated because the rest of its offering is strong.
Three states that are never allowed to collapse
Most of the honesty in this rating lives in one distinction, so it is enforced by the type system rather than left to care:
- Not assessedWe have not looked at this dimension yet, or the vendor is not testable on it. This is never scored as zero and never counted against the vendor.
- UnavailableA published finding: we looked, and there was not enough public signal to rate. It is an absence of evidence, not a low position, and is not sorted below Bronze.
- TrailsA real, sourced finding that the vendor is behind its peers on this dimension.
Citation discipline
Every finding carries a source and a retrieval date, and every reference records how well it was actually checked. Confirmed means the primary text was read directly. Search-corroborated means it was consistent across sources but the primary was not read. Unverified means it is a named lead that has not been checked, and it is never treated as a citation. A reference is not upgraded without someone reading the source.
A circular citation does not count. A vendor citing a document that itself cites the vendor is not independent validation, and is recorded as what it is.
Versioning and cadence
Every rating is stamped with the methodology version it was made under, so a rating and the bar it was judged against stay attached to each other. Full re-evaluation runs annually, with cheaper incremental updates in between when a vendor is added or a material fact changes. An incremental update is labelled as one and never presented as a full re-test.
Corrections and right of reply
A rated party may dispute any finding. A dispute we accept is published on the rating it concerns rather than resolved quietly, and a correction is marked as a correction. The full commitments are on the rated-parties policy page.