Fitness functions¶
The checks that keep humpyard's performance and architecture where the budgets and the
invariants say they are, and when each one runs. It is to performance what the
threat model is to security: a register of what can go wrong and what watches for it. The
machine-readable form is fitness/register.toml; scripts/select-checks.py reads it to pick the checks a change
needs, and tests/architecture.rs fails when this page, the register and src/ disagree.
Why not run everything every time¶
The isolation tests take minutes in release mode on a quiet machine, and the absolute latency budgets and the
throughput comparison cannot be run on a shared runner at all: its noise is as large as the tolerance. So a check
runs when a change can move it, every week on main (a path map misses a dependency's behaviour and drift made of
many small changes), on demand, and at release. A regression found by the schedule instead of the path map is a
reason to widen that check's triggers, recorded in the register.
Module classes¶
Every module under src/ has a class in the register, and a module without one fails the architecture test.
| Class | Meaning | Modules |
|---|---|---|
| request | On the path of a request | auth, budget, clock, error, estimate, health, latency, limiter, metering, metrics, num, policy, pool, pricing, repair, repair_request, retry_after, routing, scrub, select, server, signals |
| shared | Shares something the request path depends on: the runtime setup, the ledger and its writer, the configuration the request path reads, the administrative handlers that read the ledger | admin, config, ledger, lib, main, overrides, quota |
| off | Never in the serving path: commands that run once and exit, and the GUI | ctl, discover, init, migrate, prices, retired, ui, ui_assets |
A request or shared module must be named by the triggers of at least one blocking check, so a module cannot join the request path without something watching it.
The checks¶
| Check | Protects | Runs when | Where | Cost | Blocks |
|---|---|---|---|---|---|
architecture-rules |
The invariants, among them 24: heavy ledger reads stay off the runtime and behind the gate | Any change to src/, Cargo.toml, its own test or this register; weekly; release |
Hosted CI, the check job |
Seconds | Yes |
isolation-admin-reads |
ADR 0036: administrative reads do not move request latency (p99 within 10% or 1 ms of the unloaded run, on one worker, no entry dropped) | A change to a request or shared module, Cargo.toml, Cargo.lock, the toolchain file, tests/isolation.rs, tests/common/ or the CI workflow; weekly; release |
Hosted CI, the isolation job |
Minutes | Yes |
isolation-prune-contention |
A large prune holds the write lock only briefly (slowest override write under 500 ms, requests unaffected) | The same | Hosted CI, the isolation job |
Minutes | Yes |
overhead-budgets |
The budgets: added latency p50 at most 2 ms and p99 at most 10 ms at 64 concurrent, and the streaming first chunk | At release | The reference machine, by hand | Minutes | Yes, at release |
memory-budget |
Resident memory under load at most 64 MiB and growth between two loads at most 8 MiB | At release | The reference machine, by hand | Minutes | Yes, at release |
throughput-comparison |
Throughput no more than 10% below the previous release, measured alongside it | At release | The reference machine, by hand | Minutes | Yes, at release |
ledger-scale |
Ledger reads and writes at 450,000 rows | At release | The reference machine, by hand | Minutes | No |
How CI uses it¶
The changes job runs scripts/select-checks.py on the files a pull request (or a push to main) changes. The
isolation job runs only when it was selected; the weekly run (Mondays), a manual dispatch and a release tag select
every hosted-CI check. The performance-gate job always runs and is the result to require: it passes when nothing
was selected or every job that ran passed, and fails when one failed or was cancelled, so a skipped heavy job never
leaves a pull request waiting. When the weekly run fails, the report-weekly-failure job opens an issue labelled
performance-regression (or comments on the open one) naming the run; it never closes it, and pull requests and
ordinary dispatches never open one. A dispatch with rehearse failure ticked forces that path to be checked.
The weekly run does not claim to cover the reference-machine checks. They are run and recorded at release (see Deployment).
Adding a module or a check¶
A new module under src/ gets a class in [modules]. A new check gets an entry in [[checks]] and a row here with
the reason it exists; if it is slow, say what can move it and let the register decide when it runs. A change that does
either updates the register in the same pull request, and the architecture test says so when it has not.