Developer workflow · Reporting
HOPE prunes MoE experts. Does that let your API use fewer GPUs?
Amazon’s HOPE walkthrough shows how pruning reduces expert weights. Compare the authors’ coding scores, then test memory and concurrency before downsizing.
A smaller checkpoint is the starting point
On November 9, Amazon researchers published a walkthrough for HOPE, a method for pruning experts from mixture-of-experts models described in their September preprint. If you serve a model behind your own API, the useful question is whether the smaller checkpoint can meet the same quality and latency targets on fewer or cheaper GPUs. The percentage of experts removed does not answer that on its own.
A mixture-of-experts layer has many candidate networks, but a router picks only a few for each token. HOPE uses calibration traces to estimate pairwise expert contributions before deciding which weights to remove. The public tool records interactions, solves for a removal set and exports a smaller checkpoint. Removing candidates changes who can be selected; with the same routing width it does not reduce how many are selected for each token.
Read both columns before choosing a pruning rate
For Qwen3.5-35B-A3B with half of the experts in each layer removed, the authors reported a 42.33% SWE-Bench Pro resolve rate for HOPE after Evol-CodeAlpaca calibration, versus 41.82% for the unpruned checkpoint. With SWE-Bench Verified trajectories used for calibration instead, HOPE scored 40.78% on the same evaluation. At that same pruning rate, REAP also differed across the calibration sets.
That is a reason to treat calibration as part of the deployment recipe. Keep the prompts used to choose experts separate from the tasks used to approve the model. For a coding API, include cases where an agent must inspect a repository, use tools and finish with passing tests; a short code-completion score can miss failures that only appear over several turns.
The same pruning rate produced different coding scores
Scroll sideways to read all columns.
| Checkpoint | Experts removed per layer | Evol-CodeAlpaca (%) | SWE-Bench Verified (%) |
|---|---|---|---|
| Unpruned | 0% | 41.82 | 41.82 |
| HOPE | 50% | 42.33 ± 1.23 | 40.78 ± 0.86 |
| REAP | 50% | 36.23 ± 0.93 | 40.21 ± 0.43 |
Price the server that meets your load target
First run the original and pruned checkpoints with the same hardware, precision, runtime settings and request mix. Record task success, peak GPU memory and the latency of slow requests at your target concurrency. Include your longest expected inputs. The server also needs memory for the attention cache, so fitting the weights at startup does not establish that it can handle production traffic. If you also quantize the weights, add a separate run: the MoE-XBench study found cases where a smaller memory footprint came with lower throughput.
Only then try the smaller deployment. Keep the same arrival pattern and count timeouts and retries, then divide the serving spend by accepted tasks over the same window. If the smaller checkpoint fits but cannot sustain your latency target, the freed memory may be more useful for concurrency on the existing server than for moving to a cheaper one. Measure that trade before changing the rental.
Sources & verification
Provider documentation supports the facts referenced below. Our explanations and recommendations are editorial interpretation.
- Amazon researchers: Your Experts Have Partners. There's HOPE for Keeping Them Together ↗Checked Nov 10, 2026
- Tseng et al.: Higher-order pruning of experts in mixture-of-experts language models ↗Checked Nov 10, 2026
- AWS Labs: HOPE implementation ↗Checked Nov 10, 2026
- Benazir et al.: Benchmarking composable compression in mixture-of-experts LLMs ↗Checked Nov 10, 2026
- vLLM: Engine arguments and cache configuration ↗Checked Nov 10, 2026
Corrections: hello@aiapicost.com