Developer workflow · Reporting

HOPE prunes MoE experts. Does that let your API use fewer GPUs?

Amazon’s HOPE walkthrough shows how pruning reduces expert weights. Compare the authors’ coding scores, then test memory and concurrency before downsizing.

A hypothetical expert layer stores eight experts before pruning and four after, with two selected per token in both cases. This is not a speed or memory measurement.
Illustrative counts for one hypothetical layer: with routing width fixed at two, removing half the stored experts does not halve the number selected per token. Diagram based on the HOPE preprint; not measured memory, compute or speed. AI API Cost · original routing illustration · Source ↗

A smaller checkpoint is the starting point

On November 9, Amazon researchers published a walkthrough for HOPE, a method for pruning experts from mixture-of-experts models described in their September preprint. If you serve a model behind your own API, the useful question is whether the smaller checkpoint can meet the same quality and latency targets on fewer or cheaper GPUs. The percentage of experts removed does not answer that on its own.

A mixture-of-experts layer has many candidate networks, but a router picks only a few for each token. HOPE uses calibration traces to estimate pairwise expert contributions before deciding which weights to remove. The public tool records interactions, solves for a removal set and exports a smaller checkpoint. Removing candidates changes who can be selected; with the same routing width it does not reduce how many are selected for each token.

Read both columns before choosing a pruning rate

For Qwen3.5-35B-A3B with half of the experts in each layer removed, the authors reported a 42.33% SWE-Bench Pro resolve rate for HOPE after Evol-CodeAlpaca calibration, versus 41.82% for the unpruned checkpoint. With SWE-Bench Verified trajectories used for calibration instead, HOPE scored 40.78% on the same evaluation. At that same pruning rate, REAP also differed across the calibration sets.

That is a reason to treat calibration as part of the deployment recipe. Keep the prompts used to choose experts separate from the tasks used to approve the model. For a coding API, include cases where an agent must inspect a repository, use tools and finish with passing tests; a short code-completion score can miss failures that only appear over several turns.

The same pruning rate produced different coding scores

Scroll sideways to read all columns.

CheckpointExperts removed per layerEvol-CodeAlpaca (%)SWE-Bench Verified (%)
Unpruned0%41.8241.82
HOPE50%42.33 ± 1.2340.78 ± 0.86
REAP50%36.23 ± 0.9340.21 ± 0.43
Authors’ SWE-Bench Pro results for Qwen3.5-35B-A3B, Table S4 of the September 16, 2026 preprint, checked November 10. Resolve rate is the mean reward across 731 instances expressed as a percentage. Pruned results show mean ± standard deviation in percentage points across three independent calibration trials. Evol-CodeAlpaca and SWE-Bench Verified trajectories are calibration sets, not the evaluation set. The unpruned reference is repeated for comparison.Source 1 ↗

Price the server that meets your load target

First run the original and pruned checkpoints with the same hardware, precision, runtime settings and request mix. Record task success, peak GPU memory and the latency of slow requests at your target concurrency. Include your longest expected inputs. The server also needs memory for the attention cache, so fitting the weights at startup does not establish that it can handle production traffic. If you also quantize the weights, add a separate run: the MoE-XBench study found cases where a smaller memory footprint came with lower throughput.

Only then try the smaller deployment. Keep the same arrival pattern and count timeouts and retries, then divide the serving spend by accepted tasks over the same window. If the smaller checkpoint fits but cannot sustain your latency target, the freed memory may be more useful for concurrency on the existing server than for moving to a cheaper one. Measure that trade before changing the rental.

Sources & verification

Provider documentation supports the facts referenced below. Our explanations and recommendations are editorial interpretation.

  1. Amazon researchers: Your Experts Have Partners. There's HOPE for Keeping Them Together ↗Checked Nov 10, 2026
  2. Tseng et al.: Higher-order pruning of experts in mixture-of-experts language models ↗Checked Nov 10, 2026
  3. AWS Labs: HOPE implementation ↗Checked Nov 10, 2026
  4. Benazir et al.: Benchmarking composable compression in mixture-of-experts LLMs ↗Checked Nov 10, 2026
  5. vLLM: Engine arguments and cache configuration ↗Checked Nov 10, 2026

Corrections: hello@aiapicost.com

Keep reading