subpoena

Alpha v0.1.0-alpha.1 Go Apache-2.0 OpenTelemetry

Your spans are testimony. Subpoena the witness.

Trace-native testing for OpenTelemetry, built for codebases where agents write the code and the tests. One assertion runs in unit tests, in integration and in production. An independent witness proves enforcement, and the suite is held to rules the author can't drift from.

$go get -tool github.com/sprigworks/subpoena/cmd/subpoena-lint

Go 1.24 or newer, inside a module. From anywhere:

$go install github.com/sprigworks/subpoena/cmd/subpoena-lint@latest
Read the source →

Exhibit A · the invariant

One string. Three courts.

An invariant is a TraceQL query, and any trace that matches it is a violation. This one says a session that was denied must never go on to execute anything:

invariants/no_exec_after_deny.traceql{ name = "agent.session.authorize" && span.authorization.allowed = false }
  >> { name = "agent.session.exec" }

The same file is evaluated in-process by your unit tests, against a real collector and Tempo in integration, and as a Grafana alert rule in production. Promoting it between tiers is a copy and paste, and a test checks that the copies still match.

TIER 1

Unit tests

An in-memory exporter captures spans. Collect() force-flushes, and errors if a span was dropped or left unfinished.

runs in make test and CI
TIER 2

Integration

Grafana OATS (OpenTelemetry Acceptance Tests) with a real collector and Tempo, plus Subpoena's own eBPF exec witness, which reports each exec's session cgroup. Argument correlation is the fallback.

runs in make oats and CI
TIER 3

Production

The same TraceQL string as a Grafana alert rule, plus a liveness rule for when authorize spans stop arriving. Running it through Grafana's scheduler found the first version could never fire.

runs in make alerts and CI

Exhibit B · testimony vs. proof

The code that enforces a policy also reports enforcing it.

If the deny check breaks, the span saying "denied" can break along with it. Asserting on spans alone asks the defendant whether they did it.

Testimony

The application's own spans: what it says it did. Subpoena treats these as first-class test data, typed by a Weaver registry so names can't drift.

Proof

A process-exec observer outside the application's control. Every session runs in its own cgroup named with its trace ID, so the witness can tell you what actually spawned under that trace.

Out-of-process observers in tests aren't new: netaudit, for one, wraps a test run in strace and checks its network egress. What Subpoena adds is the join: witness observations keyed to a trace ID, so a test can ask what happened under this trace, and absence checks that fail when capture was incomplete.

examples/gateway/gateway_test.go// (c) the invariant returns zero traces.
harness.RequireAbsent(t, tr, invariants.MustLoad(invariants.NoExecAfterDeny))

// (e) the independent witness saw zero spawned processes for this trace.
if spawned := r.w.Spawned(ids[0]); len(spawned) != 0 {
	t.Fatalf("witness saw processes after a denial: %+v", spawned)
}

Absence is the hard case. "We saw nothing" proves nothing if the capture was incomplete, so absence assertions return Pass, Fail or Inconclusive, and Inconclusive fails by default.

Exhibit C · cross-examination

Prove the suite can tell the difference.

Functions marked //subpoena:enforcement_point get cross-examined: subpoena-cross removes each enforcement statement and reruns the suite. If the tests still pass, they weren't testing enforcement.

Authorization code has been mutation-tested before (AutoMutSec). The difference here is that the mutant leaves the record intact: the authorize span still says allowed = false while the action goes through. A suite that only checks the decision span stays green.

$ go run ./cmd/subpoena-cross ./examples/gateway
baseline: go test ./examples/gateway/...
mutant: Authorize (examples/gateway/gateway.go:145) removed "return ctx, fmt.Errorf(...ErrDenied...)"
enforcement-mutant-killed Authorize
  suite went red; failing: TestDeniedSessionCgroupStaysEmpty,
    TestDeniedSessionNeverExecs, TestHTTPDenyIs403AndSpawnsNothing

The rest of the rules are the repository's AGENTS.md, the instructions coding agents read before they touch the code, and subpoena-lint checks each one on every change. An agent working in the codebase inherits them rather than being trusted to remember them. The lint checks that the markers and calls are there, not that they're honest; cross-examination is what tests the tests.

  1. No telemetry the registry doesn't know. A span name or attribute key that isn't in the registry is flagged; use the generated constants.
  2. Collect before you assert. A test file that never calls Collect() is flagged, and so is an absence check with no Collect() before it.
  3. Mark provenance. Each test file must declare spec_derived (written from the registry and docs) or impl_derived (written after reading the code).
  4. Enforcement points need a witness. A function marked as enforcing policy is flagged unless some test asks the witness what actually happened.
  5. Take the tracer from the harness. Code under test that builds or installs its own tracer provider is flagged: spans sent elsewhere make every absence check pass for the wrong reason.
  6. The registry is the contract. A change set that touches the registry and the tests asserting on it is flagged for review, and CI blocks breaking changes without an approval label.

Exhibit D · the record

What runs today, and where it was checked.

Subpoena is in alpha and Go-only for now. Python and TypeScript are next, and the contract a port must meet, with a check for each part, already exists. The API will change before 1.0. Every kernel-level tier below (cgroups, the eBPF witness, OATS, the alert rules, Tempo conformance) runs in GitHub CI on every pull request.

PieceStatus
Registry, Weaver check, Go generationruns Weaver 0.28
pkg/harness, pkg/traceql, pkg/witness, pkg/cgroupruns in make test
subpoena-lint, subpoena-crossruns gateway mutant killed under cross-examination
eBPF exec witness, pkg/witness/execbpfruns against the real kernel in make test-ebpf
Tier 2, cgroup-correlated witnessruns a pass, and a fail against the mutant
Tier 2, argument-correlated witnessruns the fallback: a pass, and a fail against the mutant
Tier 3, Grafana alert rulesruns Grafana 13.2 and Tempo 3.1: quiet on no traffic and on clean traffic, fires on one violation
Test-only attributes kept out of production, pkg/productionruns Go exporter, and a generated Collector config checked on otelcol-contrib 0.116.1
Adopting from outside, examples/adopterruns a separate module, built and tested in CI. Migration guide
TraceQL subset vs. real Temporuns 0 divergences, pinned to Tempo 2.8.2 for the subset check, across 9 fixtures and 13 queries
Port checks, subpoena-traceql, subpoena-conformruns a port's evaluator against 139 cases: 13 queries over 9 fixtures, plus 22 queries it must reject
Python and TypeScript names, generated/python, generated/tsruns generated by Weaver from the same registry, compiled in CI; a test checks Go, Python and TypeScript agree
Harness rules as data, conformance/harness.yamlruns Collect, completeness and the three verdicts as cases every port's harness must pass; the Go harness passes them
Python and TypeScript harnessesnot built; the contract and its checks exist: names, evaluator, harness cases, witness format

The TraceQL evaluator is a documented subset rather than Tempo's engine, by design: Tempo is AGPL-3.0 and Subpoena is Apache-2.0. A conformance suite runs every fixture through both and requires identical verdicts.

Array attributes are refused, not guessed. We measured Tempo 2.8.2: one condition on an array means "some element matches", but combinations don't follow from that rule. So a query that reads an array attribute fails the test, rather than risk a tier-1 pass that tier 3 would contradict. The measurements are in the subset docs.