Inspect AI
Open-source framework for LLM and agent evaluations
Inspect is an open-source framework for frontier AI evaluations, developed by the UK AI Security Institute and Meridian Labs. Evaluations are built from composable datasets, agents, tools and scorers, and the project ships more than 200 pre-built evaluations that run on any supported model. It supports agent evaluations with built-in agents, multi-agent primitives and the ability to run external agents such as Claude Code, Codex CLI and Gemini CLI. Tool calling covers custom and MCP tools plus built-in bash, Python, text editing, web search, web browsing and computer tools, and untrusted model code can run in sandboxes on Docker, Kubernetes, Modal, Proxmox, Vagrant and others. The Inspect View web tool and a VS Code extension help monitor and debug runs. Install with pip install inspect-ai.
Use Cases
- Write behavioral tests for your own agents and run them on each release
- Compare models on the same task set before switching providers
- Run external coding agents inside a sandbox and score the result
- Run published benchmarks from the pre-built evaluation set
- Review logs of each run in Inspect View
Key Features
Composable building blocks
Datasets, solvers, agents, tools and scorers that can be reused across evaluations.
Pre-built evaluations
Over 200 benchmark implementations listed in the Evals section of the docs.
Agent evaluations
Built-in agents such as ReAct, multi-agent primitives, and an agent bridge for running external agents.
Sandboxing
Run untrusted model code in Docker, Kubernetes, Modal, Proxmox, Vagrant and other systems through an extension API.
Tool approval
The user guide documents tool approval and tool result review for agent runs.