Architecture › Tests, benchmarks and CI

Tests, benchmarks and CI

Doberman is a security tool, so its test suite carries more weight than most. This page covers how the suite is organized, what CI checks on every pull request, and the honesty rules that keep published numbers meaningful.

Three tiers#

Unit tests (tests/unit/) are deterministic: no real network, clock, or secrets. Anything time- or secret-dependent is injected as a fixture so the same run always produces the same result.

Integration tests (tests/integration/) exercise the full path against fake downstream servers instead of real tools. The standing check here is behavioral, not just structural: a BLOCK verdict must mean the fake server recorded nothing at all, not just that Doberman said no.

Redteam tests (tests/redteam/) run attack corpora through the engine and score the result: attack success rate (ASR) benchmarks, poisoning runners that try to teach a baseline to wave through something it shouldn't, and parity runners that check a guarantee holds the same way across hosts. Fixtures include adversarial suffix corpora built for these attacks specifically.

flowchart LR
  subgraph SUITES["Test tiers"]
    U["unit<br/>deterministic, seeded"]
    I["integration<br/>fake tool servers ·<br/>real plugin install discovery"]
    R["redteam<br/>attack corpus · ASR ·<br/>poisoning · parity runners"]
  end
  subgraph CI3["CI (every PR)"]
    M["test matrix<br/>ubuntu 3.11 / 3.12 / 3.13<br/>windows 3.12"]
    PS2["package-smoke-test"]
    SS["secret-scan<br/>gitleaks, full history"]
    IL["import-linter contracts"]
    SA["standalone guarantee<br/>(no plugins installed)"]
  end
  SUITES --> CI3
  BM["benchmark gate<br/>shadow → measure → graduate<br/>(honesty rules)"] --- R
  PM["parity matrix<br/>guarantee × host"] --- R

Raise-side changes ship shadow-first and graduate on measured evidence; published numbers stay subordinate to the honesty rules below.

What CI checks on every pull request#

The test job runs on Ubuntu across Python 3.11, 3.12, and 3.13, plus one Windows leg on 3.12. Doberman has OS-specific code, a Windows key-write path in storage, for instance, so that leg exists to catch platform regressions before a contributor's machine does. Inside the job: ruff check and ruff format --check, an offline markdown link check, lint-imports to enforce the import-linter contracts described in the system atlas, the pytest suite itself with a coverage floor, and a check that the parity matrix is current.

A separate package-smoke-test job builds the wheel, installs it into a clean virtual environment, and confirms the CLI entry point runs and the bundled package data (built-in roles) actually shipped. That catches packaging mistakes unit tests can't, since unit tests run against the source tree, not the built artifact.

A separate secret-scan job runs gitleaks across the repository's full git history on every PR, not only the diff.

Inside the same test job, a standalone-guarantee check installs core and runs its checks with no plugin package present at all. The extension seams described in the system atlas are additive, never load-bearing for the base guarantee.

The coverage floor is raise-only#

The pytest run enforces a coverage floor. Like every other guardrail in this project, that floor only moves up: a pull request that would need to lower it to pass gets its coverage gap closed instead, never the bar itself.

Honesty rules for published numbers#

Benchmarks measure Doberman as a decision function over labeled tool-call cases: no live LLM in the loop, so results are deterministic and reproducible from the same corpus. Attack success rate (ASR) counts a hard BLOCK as stopped; a stricter variant counts only hard blocks, because an AUTH still depends on a human answering correctly. The corpora are not filtered down to cases the engine wins — a documented non-detection stays documented rather than getting quietly dropped from the suite.

The full numbers, methodology, and known gaps live on the benchmarks page.