Open methodology

The AI model test protocol.

By Daniel Reeve · tested and updated September 2026

Twenty fixed prompts, six models, one rubric, re-run every quarter. This page publishes the method before the results, so you can decide whether the test is worth trusting before any number arrives to persuade you.

How we compare tools Browse all guides

StatusProtocol published · first run not yet published

Scope20 prompts × 6 models × 6 scored dimensions

CadenceQuarterly, with every prior run kept online

Why this page exists first

A comparison site that publishes rankings without publishing its method is asking for trust it has not earned.

Most “we tested every AI tool” articles were not tested. There is no prompt set, no rubric, no date, and no way for a reader to repeat the work and get the same answer. We would rather show the instrument than the verdict, and we would rather publish an empty results table with a date on it than a full one we cannot defend.

Everything below is fixed before any model is run. The prompts do not change between quarters; if one has to change, the change is logged on this page and the affected scores are marked as not comparable.

The instrument

The twenty prompts.

Four prompts in each of five categories. Each is a real task rather than a puzzle, because the thing worth measuring is how much editing an answer needs before it can be used.

Category	Prompts
Writing	• Rewrite this 200-word paragraph for a confident but natural business tone. Keep the meaning, remove filler, add one concrete example. • Draft a 150-word decline email to a client whose scope request is out of budget, without sounding defensive. • Summarise this 2,000-word article into five bullets a busy executive could act on. • Take this draft and tell me the three weakest sentences and why, before rewriting anything.
Reasoning	• Here are four options with costs and constraints. Recommend one, state the assumption your recommendation depends on, and name what would change your mind. • This argument has a flaw. Find it, name the flaw type, and rebuild the argument without it. • Estimate the order of magnitude of this quantity from first principles and show every assumption. • Two sources disagree on the same fact. Explain how a reader could determine which is right.
Coding	• Here is a failing function and its stack trace. Explain the bug, propose the smallest fix and add tests. • Review this diff for correctness, security and backward compatibility. Rank the issues by severity. • Refactor this 120-line function without changing behaviour, and list what could break. • Write the migration script for this schema change, including the rollback path.
Research	• What is the current published price of this product? Cite the vendor's own page and quote the figure. • Find three primary sources on this claim and state where they disagree. • This statistic is widely repeated. Trace it to the original source and report what the original actually measured. • Summarise the state of this question and mark each claim as well established, contested or unknown.
Files and long context	• From this 40-page PDF, extract every figure with its page number. • Compare these two contracts and list only the clauses that differ materially. • This spreadsheet has inconsistent headers. Describe the problems before fixing anything. • Answer this question from the attached document only, and say so if the document does not contain the answer.

The rubric

Six dimensions, scored 1 to 5.

Each answer is scored blind — model names are stripped before scoring — and every score carries a one-line justification that is published with it.

Dimension	What earns a high score
Correctness	Is it factually right? Any invented API, statistic, citation or quotation scores zero for the prompt, regardless of how well written the rest is.
Specificity	Does the answer commit to something checkable, or hedge into generality? “It depends” without saying on what scores 1.
Instruction fidelity	Did it do what was asked — including the constraint buried in the last clause — or what it assumed was meant?
Verifiability	Can a reader trace the claims? Citations that resolve and support the sentence score full marks; citations that merely exist do not.
Edit distance	How much work is left before this could be sent or merged? Measured in minutes of human editing, then mapped to the 1–5 scale.
Failure honesty	When the model does not know, does it say so? Confident wrongness is penalised harder than an admitted gap.

Recording rules

What gets published with every run.

The raw output of all 120 answers, unedited, including the bad ones. The exact model version string each answer came from, because “Claude” and “Gemini” are product names, not models. The date and time of the run, since these systems change weekly. The scorer's one-line justification per dimension. And a list of every prompt where the result was too close to separate, rather than manufacturing a winner.

Runs are executed side by side in a multi-model workspace so that all six models receive an identical prompt in the same session, which removes the most common source of noise in comparisons of this kind — prompts that drift as the tester retypes them.

Results

Q3 2026 run: not yet published.

This section will hold the scored results, the raw answers and the version strings once the first run is complete. Until then it stays empty on purpose. An empty table with a date is more honest than a full one without a method, and you can hold us to the gap.

Run	Date	Models	Status
Q3 2026	—	6	Protocol fixed 2026-09-10. Run pending.

FAQ

Questions about the method.

Why publish the protocol before the results?

Because a method disclosed after the fact can always be shaped to fit the conclusion. Fixing the prompts and the rubric first, in public, removes that option from us — and gives you something to check the eventual results against.

Why only twenty prompts?

Because twenty prompts scored carefully and published in full is worth more than two hundred scored casually and summarised. The limit is honest scoring capacity, not ambition.

Why is scoring blind?

Because knowing which model produced an answer changes how it reads. Model names are stripped before scoring and reattached afterwards.

Will you publish results that contradict your own recommendations?

Yes, and those are the results worth publishing. If the data disagrees with a recommendation elsewhere on this site, the recommendation changes and the change is dated.

Can I repeat this test myself?

That is the point of publishing it. The prompts are above, the rubric is above, and the recording rules are above. If your results differ from ours, we would rather hear about it than not.