# Tests, benchmarks and CI

Doberman is a security tool, so its test suite carries more weight than most. This page covers how the suite is organized, what CI checks on every pull request, and the honesty rules that keep published numbers meaningful.

## Three tiers

**Unit tests** (`tests/unit/`) are deterministic: no real network, clock, or secrets. Anything time- or secret-dependent is injected as a fixture so the same run always produces the same result.

**Integration tests** (`tests/integration/`) exercise the full path against fake downstream servers instead of real tools. The standing check here is behavioral, not just structural: a `BLOCK` verdict must mean the fake server recorded nothing at all, not just that Doberman said no.

**Redteam tests** (`tests/redteam/`) run attack corpora through the engine and score the result: attack success rate (ASR) benchmarks, poisoning runners that try to teach a baseline to wave through something it shouldn't, and parity runners that check a guarantee holds the same way across hosts. Fixtures include adversarial suffix corpora built for these attacks specifically.

```mermaid
flowchart LR
  subgraph SUITES["Test tiers"]
    U["unit<br/>deterministic, seeded"]
    I["integration<br/>fake tool servers ·<br/>real plugin install discovery"]
    R["redteam<br/>attack corpus · ASR ·<br/>poisoning · parity runners"]
  end
  subgraph CI3["CI (every PR)"]
    M["test matrix<br/>ubuntu 3.11 / 3.12 / 3.13<br/>windows 3.12"]
    PS2["package-smoke-test"]
    SS["secret-scan<br/>gitleaks, full history"]
    IL["import-linter contracts"]
    SA["standalone guarantee<br/>(no plugins installed)"]
  end
  SUITES --> CI3
  BM["benchmark gate<br/>shadow → measure → graduate<br/>(honesty rules)"] --- R
  PM["parity matrix<br/>guarantee × host"] --- R
```

Raise-side changes ship shadow-first and graduate on measured evidence; published numbers stay subordinate to the honesty rules below.

## What CI checks on every pull request

The `test` job runs on Ubuntu across Python 3.11, 3.12, and 3.13, plus one Windows leg on 3.12. Doberman has OS-specific code, a Windows key-write path in storage, for instance, so that leg exists to catch platform regressions before a contributor's machine does. Inside the job: `ruff check` and `ruff format --check`, an offline markdown link check, `lint-imports` to enforce the import-linter contracts described in [the system atlas](https://docs.trydoberman.dev/architecture/atlas/), the pytest suite itself with a coverage floor, and a check that the parity matrix is current.

A separate `package-smoke-test` job builds the wheel, installs it into a clean virtual environment, and confirms the CLI entry point runs and the bundled package data (built-in roles) actually shipped. That catches packaging mistakes unit tests can't, since unit tests run against the source tree, not the built artifact.

A separate `secret-scan` job runs gitleaks across the repository's full git history on every PR, not only the diff.

Inside the same `test` job, a standalone-guarantee check installs core and runs its checks with no plugin package present at all. The extension seams described in [the system atlas](https://docs.trydoberman.dev/architecture/atlas/) are additive, never load-bearing for the base guarantee.

## The coverage floor is raise-only

The pytest run enforces a coverage floor. Like every other guardrail in this project, that floor only moves up: a pull request that would need to lower it to pass gets its coverage gap closed instead, never the bar itself.

## Honesty rules for published numbers

Benchmarks measure Doberman as a decision function over labeled tool-call cases: no live LLM in the loop, so results are deterministic and reproducible from the same corpus. Attack success rate (ASR) counts a hard `BLOCK` as stopped; a stricter variant counts only hard blocks, because an `AUTH` still depends on a human answering correctly. The corpora are not filtered down to cases the engine wins — a documented non-detection stays documented rather than getting quietly dropped from the suite.

> **Never** claim a rule, detector, or benchmark result is airtight. Every number here describes defense-in-depth, not a guarantee against every attack shape.

The full numbers, methodology, and known gaps live on [the benchmarks page](https://docs.trydoberman.dev/architecture/benchmarks/).