AgentShield Benchmark — Agent Contributor Guide

# AgentShield Benchmark — Agent Contributor Guide

This file is read automatically by Claude Code and compatible AI coding agents.
Read it fully before writing any code or creating any PR.

## What This Repo Is

An open benchmark for AI agent security tools. It runs 537 test cases across 7 attack categories against provider adapters and publishes a scored leaderboard.

- **537 test cases** in `corpus/` (JSONL format, 7 categories)
- **Provider adapters** in `src/adapters/` (TypeScript, one file per provider)
- **Leaderboard** lives in 3 places: `README.md`, `docs/data.json`, `docs/index.html`
- **Results JSONs** are generated by CI — never committed manually

## Key Commands

```bash
npm run ci               # typecheck + lint + format + test (run before any PR)
npm run validate-corpus  # verify corpus integrity and hash
npm run benchmark        # run full 537-case suite (requires provider env vars)
npm test                 # unit tests only
```

## Hard Rules — Never Do These

Violating any of these will cause your PR to be rejected without review.

**Never commit files to `results/`.**
The CI runner generates all result JSONs after an adapter is merged and independently verified. Submitting pre-built results defeats the integrity model.

**Never add benchmark-signaling headers or fields.**
Do not include `x-test-mode`, `x-benchmark`, `x-evaluation`, or any header/query/body field that signals this is a benchmark call. The adapter must hit the exact same API surface production users hit — no exceptions.

**Never set `PROVIDER_ENABLED=true` in `.env.example`.**
All provider adapters must be opt-in. Use a placeholder value or leave it commented out. Contributors must consciously enable a provider before any data leaves their machine.

**Never introduce new npm dependencies without prior issue discussion.**
Use only what is already in `package.json`. If you believe a dependency is necessary, open an issue first.

**Never submit a PR with non-ASCII characters in source files.**
Run this check before committing — zero output means clean:
```bash
grep -rPn '[\x80-\xFF]' src/ .env.example
```
Hidden Unicode and bidirectional control characters are a security risk (Trojan Source). PRs that fail this check are not reviewed.

**Never submit a score claim that has not been independently verified.**
If you include score numbers in your PR description or README update, they must match a result generated by this repo's runner. A score produced by a misconfigured environment (wrong feature flags, wrong model tier, missing env vars) is not a valid benchmark result.

## Hard Rules — Always Do These

**Always wrap outbound HTTP calls in an AbortController with a ≤30s timeout.**
A non-responsive provider must not hang the benchmark runner.
```typescript
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 30_000);
const response = await fetch(url, { signal: controller.signal });
clearTimeout(timeout);
```

**Always handle errors gracefully.**
Non-2xx responses, malformed JSON, network errors, and timeouts must return a failed evaluation result — never throw an uncaught exception that crashes the runner.

**Always verify your provider configuration before running the full suite.**
Check that all required env vars are set and that the provider is responding correctly on a single test case before running all 537. A missing feature flag or wrong tier setting will produce a score that does not represent the product.

**Always update all three leaderboard locations together.**
When a score changes, these three files must be updated in the same commit:
1. `README.md` — provider table (re-sort by rank if needed)
2. `docs/data.json` — structured score data
3. `docs/index.html` — rendered leaderboard

Updating one or two but not all three leaves the leaderboard in an inconsistent state.

**Always run `npm run ci` before opening a PR.**
This runs typecheck, lint, format check, and tests. A PR that fails CI will not be reviewed.

## Adding a New Provider — Step by Step

1. **Open an issue first.** Title: `[Provider] <Name>`. Describe the product, the API endpoint(s) the adapter will call, and authentication requirements. Wait for acknowledgment before submitting a PR.

2. **Create the adapter file** at `src/adapters/<provider-name>.ts`.
   - Extend `BaseAdapter` from `./base`
   - Implement `setup()`, `evaluateImpl()`, `teardown()`
   - Gate activation on an env var: `if (!process.env.PROVIDER_ENABLED) return`
   - See `src/adapters/README.md` for the full interface and an annotated example

3. **Register in `src/providers.ts`** — add your adapter to the provider list.

4. **Add to `.env.example`** — use a placeholder URL, set `ENABLED` to empty or false:
   ```
   # MYPROVIDER_ENABLED=        # set to "true" to run
   # MYPROVIDER_API_KEY=        # your API key
   # MYPROVIDER_URL=https://your-instance.example.com
   ```

5. **PR scope — exactly these files only:**
   - `src/adapters/<provider-name>.ts`
   - `src/providers.ts`
   - `.env.example`

   Do not include `results/` files, README score updates, or leaderboard changes. Those happen after CI independently runs and verifies the score.

## Updating an Existing Provider's Score

When a provider version upgrade or configuration change improves a score:

1. Update `README.md` — edit the provider row in the leaderboard table, re-sort by rank if the position changes
2. Update `docs/data.json` — update the provider entry with new scores and date
3. Update `docs/index.html` — update the rendered leaderboard row

Include in your PR description:
- The version or configuration that changed
- A before/after score table (all categories)
- The corpus hash your run was verified against (from `npm run validate-corpus`)

## PR Description Template

Use this format for all benchmark-related PRs:

```markdown
## Summary
- <one-line description of what changed and why>

## Results

| Category          | Before | After |
|-------------------|--------|-------|
| Overall Score     | X.X    | Y.Y   |
| Prompt Injection  | X.X%   | Y.Y%  |
| Jailbreak         | X.X%   | Y.Y%  |
| Data Exfiltration | X.X%   | Y.Y%  |
| Tool Abuse        | X.X%   | Y.Y%  |
| Over-Refusal      | X.X%   | Y.Y%  |
| Multi-Agent       | X.X%   | Y.Y%  |
| Provenance        | X.X%   | Y.Y%  |
| P50 latency       | Xms    | Yms   |

## Checklist
- [ ] `npm run ci` passes locally
- [ ] Corpus hash verified: `<hash from npm run validate-corpus>`
- [ ] No files in `results/`
- [ ] ASCII-only source (`grep -rPn '[\x80-\xFF]' src/` returns nothing)
- [ ] AbortController timeout ≤30s implemented
- [ ] All three leaderboard locations updated: README.md, docs/data.json, docs/index.html
```

## Corpus Changes

If you are adding or modifying test cases in `corpus/`:

- Run `npm run validate-corpus` after every change — it checks schema, uniqueness, and category membership
- The corpus hash will change; note the new hash in your PR
- All test case IDs must be unique across the entire corpus
- `expected_behavior` must be either `"block"` or `"allow"` — no other values

## File Size Awareness

The `results/` directory contains large JSON files (tens of MB for full runs). Never commit these — they belong in CI artifacts, not the repo. If you see a large file in your diff, stop and remove it before committing.

## Questions

Open an issue or see [CONTRIBUTING.md](CONTRIBUTING.md) for the full policy.