METR Inspect Hawk
Inspect AI evaluations at scale on isolated infrastructure
Inspect Hawk is METR’s open-source platform for running Inspect AI evaluations on cloud infrastructure (documented AWS path: isolated Kubernetes pods, managed LLM proxy, PostgreSQL warehouse, web UI). You define tasks, agents, and models in YAML; Hawk provisions runners, streams logs, stores results, and can run Inspect Scout scans over transcripts. Inspect AI itself is the UK AI Security Institute evaluation framework; Hawk is the scale-out harness. METR (metr.org/about) evaluates frontier models’ autonomous capabilities and publishes methods; it is funded by donations and states it has not accepted funding from AI companies. Use Hawk as the acceptance-test runner in the Harness: Permission Outside the Agent playbook (behavioral evals before widening egress auto-allow).
Use Cases
- Run Inspect AI eval sets across models and agents on isolated pods
- Treat egress-allowlist and tool-policy suites as release gates
- Store eval logs in a warehouse and compare across runs
- Scan completed transcripts with Inspect Scout
- Keep eval infrastructure separate from the production agent cell
Key Features
Inspect AI at scale
Inspect provides tasks, solvers, scorers, and sandboxes. Hawk runs the grid of tasks × agents × models without hand-provisioning machines.
Isolated runners
Documented architecture: EKS pods for eval compute, ECS for API/LLM proxy, Aurora PostgreSQL for results, S3 for Inspect logs.
Managed LLM proxy (Middleman)
Optional proxy for provider calls with access control. You can disable it and pass your own keys.
Inspect Scout scans
Automated scanners over completed transcripts (e.g. reward hacking, safety-relevant behavior) producing per-sample scan results.
Open source
github.com/METR/hawk. Docs at hawk.metr.org. Self-host; AWS usage is on the operator.