Methodology · v1.0 · Jun 2026
REAP·EVAL
A reusable, model-agnostic protocol for judging whether a pruned or compressed Mixture-of-Experts checkpoint is worth running.
Rubric · thresholds · tiers
11 red flags
4 behavioral tests
scorecard + checklist
FILE 01 / 14 · THESIS
The premise
A pruned MoE is guilty until proven innocent.
The burden of proof is base-vs-pruned quality benchmarks on agentic / coding tasks — posted by the producer, or run by you.
Download counts, throughput numbers, observation dumps, and a clean-looking name are not evidence of quality.
FILE 02 / 14 · CONTEXT
Why it's hard to judge
Know what pruning touches
Changes
- routed-expert count
- expert tensors & ID mapping
- checkpoint size
- memory footprint
Preserves (by design)
- context length · tokenizer
- attention architecture · layers
- experts active / token
- chat format
So fits on 1 GPU, 200K context, fast prefill are preserved almost for free — they are fit / throughput facts, not quality. Never let them stand in for quality.
FILE 03 / 14 · WORKFLOW
The six-step pass
How to run an evaluation
- Identify — base, prune ratio, quant, size reduction.
- Read the card — find the disclosed tradeoff & every warning word.
- Verify provenance — are cited datasets public & real? (401/404)
- Classify evidence — tier-weight, confidence, relative drop.
- Separate prune vs quant — attribute the degradation.
- Test it yourself — loops, bounded vs open, sweep, needle.
FILE 04 / 14 · METRIC
The metric rule
Always relative drop
Percentage points mislead. 80%→73% is 7 pp — but 8.75% relative. The relative figure is what crosses thresholds.
<5%Minimal
5–15%Marginal
>15%Failure
…or any repetition / collapse / “do not use / not stable” — regardless of the number.
FILE 05 / 14 · WEIGHTING
Benchmark weighting
Not all benchmarks count equally
TIER 1SWE-Bench Verified · Terminal-Bench · browser-use · HumanEval+/MBPP · LiveCodeBench — agentic/coding, highest weight
TIER 2GSM8K · MATH · GPQA · BBH — reasoning/math
TIER 3MMLU · ARC · HellaSwag · knowledge QA — cheap to preserve, least diagnostic
Tier-1 veto: a model cannot be a Clear Win with >10% relative drop on any Tier-1 benchmark — no matter how good Tier-3 looks.
FILE 06 / 14 · CONFIDENCE
Confidence levels
How much to trust the numbers
High
- baseline posted
- n≥100 coding · ≥250 MMLU
- lm-eval / multiple evals
Medium
- proxy n=50–100
- self-reported
- partial baseline
Low
- qualitative only
- no baseline · n<50
- “experimental” · private
Sample-size floor: discard any <50-sample proxy or any “97% retained”-style claim with no baseline.
FILE 07 / 14 · TELLS
Red-flag catalog
Eleven ways a lossy prune hides
RF-01
Placeholder benchmark tables
literal {{VAR}} / TBD cells under a “Benchmarks” heading
RF-02
“Pruned / %” scrubbed from name
reads like a first-party release; tradeoff only in body
RF-03
Buried warning words
experimental · alpha · not stable · do not use
RF-04
Interrupted / debug-only bench
“not a final score” · artifact ≠ measurement
RF-05
Private / dangling evidence
cited dataset returns 401 / 404 → unverifiable
RF-06
Unbacked retention %
“97.9% retained” with no baseline or sample size
RF-07
Citation drift
bibtex doesn’t match the arXiv ID it cites
RF-08
Throughput as quality
only tok/s · TTFT · context · “fits on 1 GPU”
RF-09
Narrow eval suite
English-only n=50; misses multilingual / long-form
RF-10
Downloads as proof
“Proven” / popularity in lieu of validation
RF-11
Hygiene inconsistency
leaked local paths / tokens → low-rigor signal
FILE 08 / 14 · PROBES
Behavioral test battery
When the card has no quality numbers — test it
★ Repetition-loop probe
Open-ended, long-form prompts, ≥2048 tok. Fail = token/phrase loops, never-terminating reasoning, no natural stop. Most important for ≥40% prunes.
Bounded vs open-ended
JSON · tools · code · short Q&A usually survive; prose breaks. Report the split, not one verdict.
Sampling sweep
temp 0 · rep_penalty 1.0→1.15. Loops persist ⇒ structural, not a sampling bug.
Long-context needle
Insert a fact at depth; verify retrieval + clean termination across context bands.
Producer-side: gate-repair audit — surviving router gates must equal source rows, max abs diff 0.0.
FILE 09 / 14 · TAXONOMY
Verdict taxonomy
Four outcomes
Clear Win
All Tier-1 drops <5%, baseline posted, n≥100, no warnings, no loops.
Marginal
Tier-1 drops 5–15%, functional, minor degradation.
Failure
Any Tier-1 >15%, OR repetition/collapse, OR “do not use”.
Uncertain / High-Risk
No baseline · n<50 · “experimental” · private evidence.
Default suspicion: ≥40% prune + open-ended generation = assume repetition-loop risk until tested.
FILE 10 / 14 · INSTRUMENT
The scorecard
One sheet per model
Evaluation scorecard§8
baseline posted?Y / N
cited datasets public?Y / N · 401?
confidenceHigh / Med / Low
Tier-1 rel. drop__% · veto?
prune vs quant__ / __ / comb.
repetition looppass / FAIL
long-ctx needlepass / FAIL
VERDICT____
templates/scorecard.md — copy, fill, nothing blank.
FILE 11 / 14 · ECONOMICS
Stopping rule
When the effort isn’t justified
- Primary: 3 fair attempts (≤30% prune, real eval) & clear-win rate <25% → stop.
- Secondary: 2 consecutive with >15% Tier-1 drop, loops, or “do not use”.
- Validated only when: ≥3 clear wins, ≥50% success, ≥1 win at 40–50%.
Worked result (22 attempts, scaled to 62 models):
0 clear wins
~23% marginal
50% prune ≈ 60% fail
→ not justified for solo/community settings.
FILE 12 / 14 · THE FLIP
Producer checklist
If you ship a pruned MoE
- base-vs-pruned Tier-1 benchmarks (baseline, n≥100, harness named)
- prune % in the model name
- public calibration + observation datasets
- repetition-loop disclosure (safe vs unsafe modes)
- prune-vs-quant separation per variant
- correct citation; no leaked paths/tokens
- recovery run? post before/after quality
Throughput is preserved for free. Ship the quality numbers.
FILE 13 / 14 · EXHIBIT A
Worked example
Kimi-K2.6-519B · 50% prune
FAILURE — general use
MARGINAL — bounded use
quality baselinenone posted
cited benchmark-traces datasetHTTP 401 — private
author warning“not stable … repetition loops on open-ended”
sampling sweep fixes loops?no — persists at temp 0, rep_pen 1.15
bounded / agenticpass
long-context needle @ 260kpass · gate audit diff 0.0
Real credit for the serving/QA engineering — an unflinching Failure on the open-ended quality claim the producer’s own warning confirms.
Use it
Ship the quality numbers.
The methodology, scorecard, producer checklist, and worked examples are open. Apply it to any pruned / compressed MoE release.
repo · github.com/DJLougen/reap-eval
MIT
contributions welcome