A Claude Code plugin marketplace holding one plugin: bench-open-model, which benchmarks open-weight Hugging Face models on AWS GPU hardware and tears the hardware down afterwards.
This plugin launches EC2 GPU instances in your AWS account. They cost roughly $2-15/hr, and in some configurations over $100/hr. Billing starts at launch and continues until the instance is terminated, whether or not anyone is watching.
Do not run this against a production account or environment. Use a dedicated sandbox, development, or test account. Specifically:
- Each run creates a CloudFormation stack owning an EC2 instance, a security group, a key pair, an IAM role and an instance profile. Teardown deletes that stack.
teardown.sh --sweep-region <region>lists orphaned stacks that your identity created and requires confirmation before deleting them. It will not touch another user's stacks or untagged resources, but it is still a delete operation.- It is a load-generation tool. Point it only at an endpoint it provisioned itself; never at a shared, staging, or production inference endpoint, since it saturates the target by design.
Teardown is automated and verified, and a SessionStart hook warns about instances left
running by an earlier session, but you remain responsible for what runs in your account.
Check your bill after a session, and see Cost safety below.
This is sample code, for non-production usage. You should work with your security and legal teams to meet your organizational security, regulatory and compliance requirements before deployment.
claude plugin marketplace add aws-samples/sample-gpu-open-weight-bench
claude plugin install bench-open-model@gpu-benchNo clone needed; the marketplace is fetched from git. To update later:
claude plugin marketplace update gpu-benchGive Claude a Hugging Face model ID and ask it to benchmark it. The plugin's skill drives a 7-phase workflow:
- Size - reads HF metadata and
config.json, computes weights + KV cache + overhead, and ranks candidate EC2 GPU instances using livedescribe-instance-types, AZ offerings, and On-Demand pricing. No hardcoded instance tables. Covers standard full attention (MHA/GQA/MQA) and fails closed rather than guessing on MLA, active sliding-window, hybrid and Mamba/SSM designs - those you size from measured GPU memory instead. - Confirm - presents instance type, region,
$/hr, and estimated total. One go-ahead. - Provision - key pair, security group locked to your public IP, one instance from the current Deep Learning Base AMI, with per-AZ and per-region capacity fallback.
- Wait - polls a health marker written by the vLLM bootstrap.
- Agree the input shape - prompt and output lengths for text, or input resolution for vision, since a number without its input shape is not useful for capacity planning.
- Sweep - concurrency sweep against
localhoston the box, then validates the result was not corrupted by caching. - Report and tear down - writes a markdown report, then removes every AWS resource and verifies nothing stray remains.
Works for anything vLLM can serve: text LLMs, MoE models, and multimodal (vision / document) models. Only the load-generation step branches on modality.
Inference servers cache prompt prefixes and image embeddings. If a load test reuses any payload, the server does a fraction of the real work and reports a number that looks plausible but can be 3x too high. In one documented case a test-data generator silently recycled its input set, and the wrong figure was published before being caught.
So the scripts refuse to run a sweep that would reuse payloads, and flag the telltale signature - TTFT falling as concurrency rises - after the fact.
The plugin's references/lessons.md documents each trap and the methodology behind it:
unique-payload discipline, the silent-DPI rasterization bug, why replicas beat
tensor-parallel for models that fit on one GPU, why KV cache usually picks your hardware,
and why prefill-bound workloads saturate early. Claude reads it when it needs to explain a
surprising result or is considering overriding a guardrail.
GPU instances cost roughly $2-15/hr, so the plugin treats teardown as part of the job:
- One CloudFormation stack owns every resource, so a single
delete-stackremoves them in the correct dependency order instead of the script sequencing deletions by hand. provision.shwrites.bench-state/<run-id>.jsonbefore callingcreate-stack, so an interrupted launch still leaves a teardown trail. Each run gets a unique id, so a second run of the same model cannot overwrite the first one's record.teardown.shdeletes the stack, then independently confirms it is gone. If it cannot confirm, it keeps the state file and exits nonzero rather than reporting success.- An on-instance watchdog self-terminates after
--max-lifetime-hours(default 8) as a backstop for a lost session. - A
SessionStarthook checks.bench-state/and warns if a previous session left an instance running, with a ready-to-run teardown command. teardown.sh --sweep-region <r>finds orphaned stacks you own, shows their age and instance ids, and asks before deleting.
- AWS credentials with permission to create CloudFormation stacks, EC2 instances, security
groups, key pairs and IAM roles (
CAPABILITY_IAM), plus G/P On-Demand vCPU quota in the target region awsCLI v2 andpython3locallyHF_TOKENexported for gated models- Benchmark dependencies (
aiohttp,pypdfium2,pillow,ocrtestdata) install on the instance automatically during bootstrap
Licensed under the MIT-0 License. See LICENSE. MIT-0 is MIT without the attribution requirement, so you can lift this code into your own product freely.
Nothing is vendored in this repository. These are fetched at runtime, into your own account, by the benchmark instance:
| Dependency | License |
|---|---|
vLLM (vllm/vllm-openai image) |
Apache-2.0 |
| aiohttp | Apache-2.0 AND MIT |
| pypdfium2 | BSD-3-Clause AND Apache-2.0 |
| Pillow | MIT-CMU |
| ocrtestdata | MIT |
transformers (optional, --tokenizer) |
Apache-2.0 |