Add TESTING.md for AI agents

[ann0see]

What is the current behaviour and why should it be changed?

Currently, agents struggle with testing - especially on macOS due to sandboxing. Moreover, we should have some more detailled testing guidelines using JSON-RPC. I'll let an agent draft an idea here which should be reviewed.

Describe possible approaches

Add TESTING.md to docs/agents

Has this feature been discussed and generally agreed?

No.

[ann0see]

Edit: I don't think this proposal is good for our usecase -
We'd probably need some session which already did successful
Testing.

--

Drafting a proposal in response to the issue, since it explicitly invites an agent to do so. No PR will be opened until a maintainer agrees (per AGENTS.md and CONTRIBUTING.md).

Proposed scope

• Location: docs/agents/TESTING.md, alongside docs/agents/COMMENTING.md and docs/agents/README.md. Loaded on demand by agents; not a top-level file.
• File is long-form; the existing one-line testing note in AGENTS.md:17 stays as the short summary.
• Equal per-platform coverage (Linux, Windows, macOS, Android/iOS) with the Linux/JACK-dummy rig and JSON-RPC as the workhorse recipes. Android/iOS sections are brief — they rely on the autobuild/CI for evidence.
• macOS sandbox caveat: the --jsonrpcsecretfile path must live somewhere the sandboxed binary can open(2) (app container, file-picker-granted path). Unsandboxed dev builds can use any path. No hard-coded bundle id in the doc.

Full proposed docs/agents/TESTING.md

# Testing Jamulus changes

Companion to `AGENTS.md` and `CONTRIBUTING.md` (read both first). This file is
the long-form testing guide; `AGENTS.md` is the short summary.

The point of testing is to produce evidence — logs, RPC responses, before/after
behaviour — that you attach to the PR. An assertion without evidence is not a
test. If you did not run or verify something, say so.

## Before you start

- Re-read `AGENTS.md` (priority order: stability > real-time safety >
  backwards-compat > maintainability > features; the Never-Do rules around the
  real-time path).
- Re-read `CONTRIBUTING.md` §Testing and §Ownership.
- The test rig described here exercises your change end-to-end. For protocol or
  audio-path changes, also see `docs/JAMULUS_PROTOCOL.md`.

## Build for testability

Replace `qmake` with `qmake-qt5` on Fedora. Run `make distclean` whenever you
change `CONFIG` flags.

### Linux

```sh
qmake "CONFIG+=headless serveronly" && make          # headless server
qmake "CONFIG+=headless" && make                     # headless client + server
qmake && make                                         # full GUI build
```

### Windows

Open `Jamulus.pro` in Qt Creator and build the kit matching your target
(release/debug). For the headless build, add `CONFIG+=headless serveronly` to
the qmake step the same way. JSON-RPC requires the build NOT to define
`NO_JSON_RPC` (see `Jamulus.pro`).

### macOS

Generate the Xcode project, then build the `Jamulus` scheme. For a headless
build, pass `CONFIG+=headless serveronly` to the qmake step. Run from the
`.app` after `macdeployqt`.

### Android / iOS

These targets are not exercised end-to-end on a developer machine. Build per
`COMPILING.md` and rely on the autobuild / CI for evidence. Document which CI
artefact you inspected in the PR.

## Stand up a local rig

Pick a free pair of ports: audio default is 22124 (`DEFAULT_PORT_NUMBER` in
`src/global.h:115`); use 22100 for JSON-RPC and 22222+ for extra clients
(`src/testbench.h:76`).

### Linux (with dummy JACK)

```sh
jackd -d dummy &
./Jamulus -s -n -p 22124 --jsonrpcport 22100 --jsonrpcsecretfile /tmp/jrpc.secret
./Jamulus -n -c localhost:22124
```

`-s` server, `-n` no GUI, `-c` connect, `-l` enable logging, `-j` disable
auto-JACK-connections when you want to wire manually.

### Windows

```sh
Jamulus.exe -s -n -p 22124 --jsonrpcport 22100 --jsonrpcsecretfile %TEMP%\jrpc.secret
Jamulus.exe -n -c localhost:22124
```

For ASIO, leave the GUI up briefly to pick a device, then restart headless.
`--nojackconnect` (-j) if you are not using JACK on Windows.

### macOS (sandboxed app bundle, the default after `macdeployqt`)

The app is sandboxed; `--jsonrpcsecretfile` must point somewhere the sandboxed
process can `open(2)`. Use the app's container:

```sh
mkdir -p "$HOME/Library/Containers/<bundle-id>/Data/tmp"
openssl rand -base64 10 > "$HOME/Library/Containers/<bundle-id>/Data/tmp/jrpc.secret"
./Jamulus.app/Contents/MacOS/Jamulus -s -n -p 22124 \
  --jsonrpcport 22100 \
  --jsonrpcsecretfile "$HOME/Library/Containers/<bundle-id>/Data/tmp/jrpc.secret"
./Jamulus.app/Contents/MacOS/Jamulus -n -c localhost:22124
```

Substitute `<bundle-id>` with the actual id of the build you are running
(visible in the `.app/Contents/Info.plist`, or in
`~/Library/Containers/` after first launch). A path outside the container
gives the error `JSON-RPC: Unable to open secret file <path>. Exiting.`
(`src/main.cpp:955`).

For an unsandboxed development build (Xcode run, or a build that skipped
entitlements), the rig is identical to Linux: `/tmp/jrpc.secret` is fine.

## Driving the rig via JSON-RPC

JSON-RPC uses newline-delimited JSON over TCP on `127.0.0.1` (configurable
with `--jsonrpcbindip`). Full method reference: `docs/JSON-RPC.md`. Secret
must be at least 16 chars (`src/main.cpp:960`).

### One-time setup

```sh
openssl rand -base64 10 > /tmp/jrpc.secret       # Linux
# or on Windows:
#   openssl rand -base64 10 > %TEMP%\jrpc.secret
# or on macOS sandboxed: see the rig above.
```

### Hand-rolled smoke test (no dependencies)

```sh
exec 3<>/dev/tcp/localhost/22100
printf '%s\n' '{"id":1,"jsonrpc":"2.0","method":"jamulus/apiAuth","params":{"secret":"'$(cat /tmp/jrpc.secret)'"}}' >&3
cat <&3 &
# ... now send more requests on fd 3, e.g.:
printf '%s\n' '{"id":2,"jsonrpc":"2.0","method":"jamulusserver/getServerProfile","params":{}}' >&3
printf '%s\n' '{"id":3,"jsonrpc":"2.0","method":"jamulusserver/getClients","params":{}}' >&3
```

`nc` (Linux) or `ncat` (Windows) work the same way. The Node `jayson` example
in `docs/JSON-RPC.md:38` is the easiest path if you already have Node.

### Pick the right method for the change

| Change touches… | Drive with | Assert |
| --- | --- | --- |
| Mute / fader / name / skill / instrument | `jamulusclient/setMuted`, `setFaderLevel`, `setName`, `setSkillLevel`, `setInstrumentCode` | `jamulusclient/clientListReceived`, `channelLevelListReceived` |
| Chat | `jamulusclient/sendChatText`, `jamulusserver/broadcastChatMessage`, `privateChatMessage` | `jamulusserver/chatMessageReceived`, `jamulusclient/chatTextReceived` |
| Server config (name / welcome / directory) | `jamulusserver/setServerName`, `setWelcomeMessage`, `setDirectory` | `getServerProfile` before/after |
| Recorder | `jamulusserver/startRecording`, `stopRecording`, `restartRecording`, `setRecordingDirectory` | `getRecorderStatus` (ack is async — see `docs/JSON-RPC.md:486`) |
| Connection lifecycle | n/a — connect/disconnect a client | `jamulusserver/clientConnected`, `clientDisconnected` |
| Version / mode | `jamulus/getMode`, `getVersion` | response payload |

### Common failures

- `No JSON-RPC support in this build.` → build defined `NO_JSON_RPC`
  (`src/main.cpp:936`). Rebuild without it.
- `Refusing to run with secret of length N (required: 16).` → regenerate the
  secret file with a longer value.
- Connection refused → wrong port, or server was not started with
  `--jsonrpcport`.

## Real-time sanity

After exercising the change, watch the server log for dropped frames, late
cycle messages, or unbounded queue growth. The Never-Do rule in `AGENTS.md`
covers this: anything that can block the audio thread (allocation, file I/O,
locks) must be moved off the real-time path. For changes in `src/sound`,
`src/socket.cpp`, or `src/server.cpp`, run the rig under load (two or more
dummy clients) and compare before/after logs.

## The testbench (fuzzing)

`src/testbench.h` fires random protocol messages at a target address — useful
for shaking the protocol parser. To activate, uncomment the two lines in
`src/main.cpp:930-933` (gated by `### TEST: BEGIN ###` markers so reviewers can
spot it):

```cpp
// CTestbench Testbench ( "127.0.0.1", DEFAULT_PORT_NUMBER );
```

Rebuild and run the rig; observe the server log for crashes, asserts, or
spew. **Always comment it back out before committing.** A testbench left
enabled is a CI-breaking change and a security smell.

## Evidence to attach to the PR

Minimum, in the PR description or as a comment:

1. The exact command lines you ran (build + rig + JSON-RPC probes).
2. The full log of one headless server run, with the lines relevant to your
   change highlighted.
3. Before/after `getClients`, `getServerProfile`, or `getRecorderStatus`
   responses if the change is observable through them.
4. The matching JSON-RPC notification (e.g. `clientConnected`,
   `chatMessageReceived`) showing the change took effect on the wire.
5. For protocol changes, a packet capture (or a transcript) and the line in
   `docs/JAMULUS_PROTOCOL.md` it implements.

If you did not run something, say so — do not assert it works.

## Reading list

- `AGENTS.md` — priority order, Never-Do rules
- `CONTRIBUTING.md` — testing paragraph, ownership, AI use
- `COMPILING.md` — per-platform build, `CONFIG` table
- `docs/JSON-RPC.md` — full method reference
- `docs/JAMULUS_PROTOCOL.md` — packet IDs, ack rules
- `docs/agents/COMMENTING.md` — when posting evidence to a thread
- `src/testbench.h` — protocol fuzzer
- `src/main.cpp:930-933` — testbench activation / deactivation point

Open questions for maintainers

1. Confirm docs/agents/TESTING.md is the right location, vs. a top-level TESTING.md or docs/TESTING.md. Current pick keeps the file alongside other on-demand agent docs and out of the human-facing top-level.
2. Equal per-platform scope vs. Linux-only is a deliberate choice. Android/iOS sections are short because the rig is desktop-only. OK to keep that, or trim those sections entirely?
3. The testbench paragraph currently lives here. src/sound/README.md has a "sound design not yet documented" TODO — is that a better home, or is it unrelated to the protocol fuzzer?

🤖 Used AI: opencode / minimax/minimax-m3:free, 2026-08-27

[ann0see]

I think the compiling part should not be duplicated - and furthermore there's no need to re read AGENTS.md or similar... I think what we really want is something like a Jamulus testing SKILL.md - definitely needs some more iteration (by humans and agents)

[ann0see]

Open Jamulus.pro in Qt Creator and build the kit matching your target
(release/debug). For the headless build, add CONFIG+=headless serveronly to
the qmake step the same way. JSON-RPC requires the build NOT to define
NO_JSON_RPC (see Jamulus.pro).

Definitely not the way to go... It should all work on the CLI.

[ann0see]

I think that we might need a larger model which already did testing write this file and plan it. I believe that we'd want to make this file a times and hints file instead. So e.g testing is always specific to the change in question. JSON-RPC methods might not exist, sometimes using tcpdump max be good etc.

[mcfnord]

My AI wants to list hints that are based on errors it made along the way. I believe the most likely error an agent will make is an error another agent already made, so accruing hints based on missteps makes sense. Per-platform guidance makes sense, but COMPILING.md is the place for the basics, with testing tips and tricks here to augment that. The tree is growing with Markdown guidance, and a big model can sort out duplications and relocations really nicely. I think COMMENTING.md may as well be a component of CONTRIBUTING.md.

[ann0see]

Yes. My rationale is to Split documentation (including Agent docs) out such that a model only needs to read those if it needs them. This would mean that this is not part of CONTRIBUTING.md. The file should probably give a structured plan on how to test features imo.

[ann0see]

Per the direction in this thread (change-specific, no duplication, CLI-only, hints from real errors), here is the current TESTING.md as code so it can be reviewed inline:

# TESTING — agent verification workflow

Read this only when you are about to test. It is written for a model that would rather not be here, so every rule is a gate against fabricated or vacuous verification, not a polite request. Run the loop below in order, per diff hunk. You are not done on a single green run.

**The design bound, stated once:** this file cannot run commands for you. The evidence is the *quoted command plus its verbatim output*, written together in the PR. A claim not backed by that pair is a self-report, not evidence. An honest "not tested: <trace>" beats a fake pass, but it is earned, not a default.

## 0. Establish the binary

A binary is required before any other step. Re-derive it and name it: `./Jamulus --version` and `./Jamulus -h` (Linux build steps are in `COMPILING.md`/`AGENTS.md`; do not repeat them here). If no binary exists and you cannot build one, stop: declare "not tested: could not build <why>" and quote the build error. Do not proceed to invent behavior for a binary that is not there.

## 1. Diff first — map every hunk

Read your diff before anything else. Enumerate each hunk that asserts behavior, and for each write a one-line mapping: *hunk → the behavior it asserts → the probe that goes red when that behavior is wrong*. A hunk with no linked probe is unverified regardless of any green run. If you cannot name that for a hunk, stop and say so rather than claiming a whole-diff pass.

## 2. Ground truth

The binary is the truth; this file is a hypothesis. Where they disagree, the binary wins. Same rule covers evidence: quote the **exact command** and its **verbatim raw output** together, so a human can re-run the command and compare byte for byte. Quote, do not describe; a paraphrase is not evidence.

## 3. The one rule — every probe must be proven able to go red

A test exists to fail. For each mapped probe, prove it can go red: invert one thing in the corresponding hunk, run the probe, show it red, then restore. Attach all three as quoted pairs — the inverted lines (or a one-line description of the inversion), the command, the verbatim red output — then the restored green. If you cannot make a probe go red, the probe is defective: fix the probe or declare that hunk "not tested: no probe could be made to fail <why>". You do not proceed to green with a probe that never can fail it.

## 4. Green is only the restored half

A green never shown to have gone red is, by definition, unverified. One green run is not done; only a red-then-green pairing, per probed hunk, moves you forward.

## 5. The failure is the finding

When a run goes red, the red is data, not an invitation to re-read your change as correct. Do not update an expectation to fit your output. You may change an expectation only if you attach the binary command and its verbatim output proving it, on the running binary, that the expectation was stale — and you must say that you did. Otherwise the red stands.

## 6. Pick the seam, and escalate

Verify at the cheapest honest boundary that exercises your diff and can go red. If the cheap seam cannot go red for your change, escalate down the stack — do not stop:
JSON-RPC → CLI black-box → socket state → wire transcript → syscall/mechanism → loss/real-time signals.

- Change in state/logic reachable over RPC → JSON-RPC seam.
- Change at network, process, or audio boundary → observation toolbox.
- When the claim is about *why* a behavior happens, prefer the seam that shows mechanism (strace, wire) over one that only shows outcome. An RPC returning the right value does not tell you why.
- Never substitute one layer's evidence for another's claim. A pcap is not the server log; a log line is not a measured underrun. Stopping at a seam that cannot fail your change is the failure.

## 7. Adversarial variant, tied to the diff

Wrongness must be the adversarial variant of the exact shape your diff changed — not an arbitrary garbage byte from the middle of wrongness. If your diff tightened a parser, hand it the malformed shape it now must reject. If you changed an ack rule, break the reverse direction. If you validated a bound, cross it. Then send the equivalent malformed input over the RPC pipe and assert the same. If it does not fail on that adversary, you have not tested the change. Do this for every mapped hunk, and attach each as a quoted command + verbatim response; the app must survive or reject cleanly.

## 8. Escape hatch, gated

Declaring "not tested" is the only way to move past something that genuinely cannot be run, and it is earned: you must name what you tried, the blocking error quoted verbatim, and why it blocks — not "no audio backend" but "no audio backend; tried `jackd -d dummy`, it failed because <quoted error>." A trace, not a reason. This is not the default state; it is the honestly-earned one, and it applies per hunk, not wholesale.

## Observation toolbox

Every technique names the file it produces, and you paste that file's raw output verbatim with the command that made it. What a technique cannot prove is part of the technique.

- **JSON-RPC** — headless server with JSON-RPC enabled, authenticate first, capture the transcript to a file. Newline-delimited JSON over TCP. Proceedings are in `docs/JSON-RPC.md`; read it only when the method catalog is needed, not for a smoke test. Proves state and wiring. Cannot show the wire bytes or audio timing.
- **CLI black-box** — run the binary, assert exit code, stdout, stderr, `-l <logfile>` server log, process uptime, socket listen. Capture each stream to a file. Proves commands exist, behave, and fail as claimed. Cannot show audio quality or timing.
- **Socket state** — `ss`/`netstat`: listener up, queues filling, peers connected (default UDP port 22124). Proves reachability and flow control. Cannot show what the app did with the data.
- **Wire transcript** — `tcpdump -i lo udp port <port>` to a pcap file, cross-referenced to `docs/JAMULUS_PROTOCOL.md` for packet semantics. Proves handshake, acknowledgements, ordering, framing. Cannot explain why the app chose what it did.
- **Syscall / mechanism** — `strace -e trace=network` (add `-T` for per-call timing): the path taken, which calls, where it blocks. The defense against outcome-match-as-proof.
- **Sustained / loss behavior** — drive server+client hard, inject loss/jitter/latency with `tc netem`, observe reconnect, recovery, crashes, leaks over time. Exercises the jitter buffer and reconnect path a one-shot run never touches.
- **OS real-time signals** — `dmesg`, ALSA/JACK xrun counters, `jack_delay` roundtrip latency. The only things that approach real-time evidence.

## When it fails

- **Server won't start:** check `--jsonrpcsecretfile` is a single-line file, secret ≥16 chars, `-s -n` present. Confirm the binary built and you are not running a stale one.
- **Client won't connect:** no audio backend. Linux: dummy Jack (`jackd -d dummy`) or `--nojackconnect`. Cross-check address and port against `-h`.
- **RPC auth fails:** secret mismatch, or a request before `jamulus/apiAuth`. Auth precedes every call.
- **No packets in `tcpdump`:** wrong interface (`-i lo`), or traffic never leaves the host. Capture while generating traffic; a capture ending in kernel drops is incomplete. If tcpdump reports `You don't have permission to perform this capture`, you lack root/CAP_NET_RAW — do not fabricate a capture. Drop to the socket-state and CLI seams (both work unprivileged), or declare the wire seam not tested with the permission error quoted as the reason.

## The honesty ceiling

State, wiring, reachability, and protocol behavior are provable here. **True real-time safety is not.** On a CI box or dummy Jack there is no meaningful underrun amplitude, so you cannot measure it — and you must not assert it. If the change touches `src/sound/`, `src/socket.cpp`, or mixing in `src/server.cpp` and you could not exercise the real audio path, quote what you tried to get closer (dummy Jack, netem, xrun counters) before conceding. Do not turn "the process stayed up" into "the audio is fine."

## Hints from real errors

Concrete and grep-able: if you hit one of these failure strings, this is the fix. Accrue here as agents hit new errors.

- tcpdump prints `You don't have permission to perform this capture`: you lack root/CAP_NET_RAW. Use `ss`/`netstat`, or say the wire seam is untested. Do not invent a pcap.
- `git submodule update --init` fails cloning `libs/oboe` (Android). Harmless on desktop builds; keep going.
- Backgrounding the headless server or jackd in a test script can hang the shell on teardown. Kill by PID (`kill -9 <pid>`), not by pattern.
- Client connects but says nothing about audio. Use `--nojackconnect` when you only need wire/log; `jackd -d dummy` when you need a Jack graph.
- Server refuses to start with JSON-RPC: `--jsonrpcsecretfile` must be single-line and >= 16 chars (`openssl rand -base64 10`). The log will not say why.
- A stale binary looks like a green pass. Re-derive with `--version` before trusting any run.
- macOS sandboxing can block the `--jsonrpcsecretfile` open path; keep it where the sandboxed binary can open(2).

## Definitions of done

A change is done when, for **every** diff hunk that asserts behavior:

- A probe is mapped to that hunk (step 1).
- That probe was shown red by inverting one thing in the hunk, then green on the restored change — both as quoted command + verbatim output (steps 3–4).
- The adversarial variant of the exact changed shape was run, with quoted response attached (step 7).
- Any expectation change carries the binary command + verbatim output proving staleness (step 5).
- The seam's ceiling is named: what it proved and what it could not (step 6, honesty ceiling).
- Anything not run is declared "not tested: <what you tried, blocking error, why>" per hunk, not wholesale.
- Where the file and the binary disagreed, the binary won and there is a quoted command showing it.

Nothing that fails these is green. A single green run, or a single quoted output, is not done.

🤖 Used AI: opencode/big-pickle, opencode harness. File drafted as the proposal the issue requested; review before any PR.

[ann0see]

It is written for a model that would rather not be here, so every rule is a gate against fabricated or vacuous verification, not a polite request.

To be removed (was just my prompting).

The design bound, stated once: this file cannot run commands for you. The evidence is the quoted command plus its verbatim output, written together in the PR. A claim not backed by that pair is a self-report, not evidence.

This sounds very much like slop tbh. I believe that the current big pickle is just not good quality enough to do concise writing tbh...

[ann0see]

Drafted with GPT.

---

TESTING — agent verification workflow

Use this workflow when testing a code change. The goal is honest, reproducible evidence, not ritual. Never claim behavior you did not actually observe.

1. Establish the binary

Before testing, verify which binary you are testing:

./Jamulus --version
./Jamulus -h

If the binary does not exist, try to build it using COMPILING.md.

If you cannot build it, stop and report:

not tested: could not build <reason>

Include the relevant command and verbatim error output.

Never infer behavior from source code when the required binary cannot be run.

2. Read the diff first

Inspect the complete diff.

For every hunk that changes behavior, identify:

• what behavior the hunk is intended to change;
• the best test/probe that exercises that behavior;
• what observable result would demonstrate failure if the change were wrong.

If a behavioral hunk has no credible probe, say so explicitly. Do not turn an untestable hunk into a fake pass.

3. Test the actual behavior

Choose the lowest seam that genuinely exercises the changed code:

• JSON-RPC for state/API behavior;
• CLI black-box testing for command-line behavior;
• socket state for connectivity/reachability;
• packet capture for protocol/wire behavior;
• syscall tracing when the mechanism itself matters;
• sustained/network-loss testing for reconnect, buffering, loss, or recovery;
• real audio/xrun measurements when making real-time audio claims.
• other appropriate tools if you cannot use the above mentioned ones

Do not use evidence from one layer to prove a claim belonging to another layer.

For every meaningful test, record:

• the exact command;
• the relevant raw output;
• the result/exit status.

Prefer capturing output to a file and quoting that output rather than paraphrasing it.

4. Test the failure mode

A test should be capable of detecting the regression it is intended to protect against.

Where practical, demonstrate this by testing the specific adversarial case corresponding to the diff:

• parser change → malformed input of the newly relevant shape;
• validation change → value just outside the accepted boundary;
• acknowledgement change → broken reverse direction;
• state-machine change → invalid/unexpected transition;
• protocol change → corresponding malformed or incompatible packet.

You do not need to mechanically mutate every line of the diff.

The important question is:

If this particular change were wrong, would this probe expose it?

If the answer is no, choose a better probe or explicitly mark that behavior as unverified.

5. Red/green testing

When useful and feasible, prove a test's sensitivity by temporarily introducing the specific regression and showing:

• changed implementation → test fails;
• restored implementation → test passes.

Include the commands and raw output.

This is especially valuable for new tests or tests whose failure sensitivity is otherwise unclear.

Do not waste time performing artificial mutations merely to satisfy a ritual when the test's ability to detect the regression is already well established.

A green result is evidence, but its strength depends on what the test actually exercises.

6. Treat failures as data

If a test fails:

• do not immediately change the expected result to make it pass;
• determine whether the failure is caused by the implementation, the test, the environment, or a stale expectation;
• if changing an expectation, provide evidence from the running binary showing why the old expectation was incorrect.

Never ever manufacture a passing result. Disclose if you cannot test something (section 9.).

7. Adversarial testing must match the change

Do not use arbitrary malformed input merely to claim "negative testing."

The adversarial input should correspond to the exact behavior changed by the diff.

For example:

• Changed: reject packet lengths > N
• Test: send a packet with length N+1
• Expected: clean rejection

Run the equivalent case through the real interface used by the application where possible.

Verify that the application either rejects the input correctly or survives it without corrupting state/crashing.

8. Know what each test proves

State the evidence ceiling.

Examples:

• A successful JSON-RPC response proves the exposed state/API behavior, not packet contents.
• A pcap proves observed wire traffic, not the internal reason for generating it.
• A log message proves the application emitted that message, not necessarily that the underlying real-time condition occurred.
• A process remaining alive does not prove audio quality.
• Dummy audio infrastructure does not establish production real-time safety.

Do not promote weak evidence into a stronger claim.

9. If something cannot be tested

"Not tested" is acceptable when genuinely necessary, but make it specific.

Report:

not tested: <behavior> — attempted <command/test>; blocked by <verbatim error>; therefore <what remains unproven>.

For example:

not tested: wire capture — attempted tcpdump -i lo ...; blocked by You don't have permission to perform this capture; therefore packet-level behavior remains unverified.

Do not simply write "couldn't test audio" or "environment limitation."

10. Final verification report

Before declaring the change tested, provide a compact mapping:

Diff hunk	Intended behavior	Probe	Result	Evidence ceiling
...	...	...	PASS / FAIL / NOT TESTED	...

For every behavioral hunk, one of these must be true:

• it has a successful probe with actual evidence;
• it has an explicitly documented failed test;
• or it is explicitly marked not tested with the attempted command, blocking error, and remaining uncertainty.

Do not call the entire change "verified" if an important behavioral hunk remains untested.

Core rule

Evidence beats assertion.

Never say "tested", "works", "verified", or "safe" merely because the code looks correct or because one unrelated test is green.

Run the narrowest honest test that can distinguish correct behavior from the relevant failure, show the actual result, and state clearly what that result does and does not prove.

[ann0see]

AI: Here is an updated draft of TESTING.md that adds repo-specific local evidence so the workflow is directly executable (build recipe, a verified JSON-RPC roundtrip, protocol-doc pointer for adversarial packets, and requiring the tested binary version in the report).

---

Testing Jamulus

Use this workflow when testing a code change. The goal is honest, reproducible evidence, not ritual. Never claim behavior you did not actually observe.

1. Establish the binary

Before testing, verify which binary you are testing:

./Jamulus --version
./Jamulus -h

If the binary does not exist, or the diff may not be represented by the binary, build it according to COMPILING.md for the target OS.

If you cannot build it, stop and report:

not tested: could not build <reason>

Include the relevant command and verbatim error output.

Never infer behavior from source code when the required binary cannot be run and claim that the fix was verified.

2. Read the diff first

Inspect the complete diff.

For every hunk that changes behavior, identify:

• what behavior the hunk is intended to change;
• the cheapest test/probe that exercises that behavior;
• what observable result would demonstrate failure if the change were wrong.

If a behavioral hunk has no credible probe, say so explicitly. Do not turn an untestable hunk into a fake pass.

3. Test the actual behavior

Choose the lowest seam that genuinely exercises the changed code:

• JSON-RPC for state/API behavior;
• CLI black-box testing for command-line behavior;
• socket state for connectivity/reachability;
• packet capture for protocol/wire behavior;
• syscall tracing when the mechanism itself matters;
• sustained/network-loss testing for reconnect, buffering, loss, or recovery;
• real audio/xrun measurements when making real-time audio claims;
• any other tool or method appropriate to proving the specific change.

Local recipe: drive state/API behavior via JSON-RPC

Start a headless server and probe it over JSON-RPC:

./Jamulus -s -n -p 22999 --jsonrpcport 22998 --jsonrpcsecretfile /path/to/secret.txt

The transport, secret-file rules, the required jamulus/apiAuth handshake, and every method are documented in docs/JSON-RPC.md. Send a minimal probe (auth + getMode + getVersion) and record the complete raw response.

Do not use evidence from one layer to prove a claim belonging to another layer.

For every meaningful test, record:

• the exact command;
• the relevant raw output;
• the result/exit status.

Prefer capturing output to a file and quoting that output rather than paraphrasing it.

4. Test the failure mode

A test must be capable of detecting the regression it is intended to protect against.

Where practical, demonstrate this by testing the specific adversarial case corresponding to the diff:

• parser change → malformed input of the newly relevant shape;
• validation change → value just outside the accepted boundary;
• acknowledgement change → broken reverse direction;
• state-machine change → invalid/unexpected transition;
• protocol change → corresponding malformed or incompatible packet.

You do not need to mechanically mutate every line of the diff.

The important question is:

If this particular change were wrong, would this probe expose it?

If the answer is no, choose a better probe or explicitly mark that behavior as unverified.

5. Red/green testing

When useful and feasible, prove a test's sensitivity by temporarily introducing the specific regression and showing:

• changed implementation → test fails
• restored implementation → test passes

Include the commands and raw output.

This is especially valuable for new tests or tests whose failure sensitivity is otherwise unclear.

Mutation is evidence about a probe's sensitivity, not a requirement to mechanically mutate every hunk.

Do not waste time performing artificial mutations merely to satisfy a ritual when the test's ability to detect the regression is already well established.

A green result is evidence, but its strength depends on what the test actually exercises.

6. Treat failures as data

If a test fails:

• do not immediately change the expected result to make it pass;
• determine whether the failure is caused by the implementation, the test, the environment, or a stale expectation;
• if changing an expectation, provide evidence from the running binary showing why the old expectation was incorrect.

Never manufacture a passing result.

7. Adversarial testing must match the change

Do not use arbitrary malformed input merely to claim "negative testing."

The adversarial input should correspond to the exact behavior changed by the diff.

For example:

• Changed: reject packet lengths > N
• Test: send a packet with length N+1
• Expected: clean rejection

To craft wire-level packets for Jamulus, see docs/JAMULUS_PROTOCOL.md for packet IDs, field layout, and acknowledgement rules.

Run the equivalent case through the real interface used by the application where possible.

Verify that the application either rejects the input correctly or survives it without corrupting state or crashing.

8. Know what each test proves

State the evidence ceiling.

Examples:

• A successful JSON-RPC response proves the exposed state/API behavior, not packet contents.
• A pcap proves observed wire traffic, not the internal reason for generating it.
• A log message proves the application emitted that message, not necessarily that the underlying real-time condition occurred.
• A process remaining alive does not prove audio quality.
• Dummy audio infrastructure does not establish production real-time safety.

Do not promote weak evidence into a stronger claim.

9. If something cannot be tested

"Not tested" is acceptable when genuinely necessary, but make it specific.

Report:

not tested: <behavior> — attempted <command/test>; blocked by <verbatim error>; therefore <what remains unproven>.

For example:

not tested: wire capture — attempted `tcpdump -i lo ...`; blocked by `You don't have permission to perform this capture`; therefore packet-level behavior remains unverified.

Do not simply write "couldn't test audio" or "environment limitation."

10. Final verification report

Before declaring the change tested, provide a compact mapping and quote the tested binary version from ./Jamulus --version so the evidence is attributable to a specific build.

Diff hunk	Intended behavior	Probe	Result	Evidence ceiling
...	...	...	PASS / FAIL / NOT TESTED	...

For every behavioral hunk, one of these must be true:

• it has a successful probe with actual evidence;
• it is explicitly documented as FAIL, with the observed evidence and diagnosis;
• or it is explicitly marked NOT TESTED with the attempted command, blocking error, and remaining uncertainty.

Do not call the entire change "verified" if an important behavioral hunk remains untested.

Core rule

Evidence beats assertion.

Never say "tested", "works", "verified", or "safe" merely because the code looks correct or because one unrelated test is green.

Run the narrowest honest test that can distinguish correct behavior from the relevant failure, show the actual result, and state clearly what that result does and does not prove.

[mcfnord]

🤖 AI: Four measured additions to the draft above, each earned by running it (Linux, and the macOS one on a Mac mini).

§1 / §10 — --version cannot attribute evidence to a build. The commit id is baked when qmake runs, not when make runs (Jamulus.pro, version block): after editing src/main.cpp, make -n re-runs qmake 0 times and the main.cpp compile line still carries the pre-edit APP_VERSION; -dirty appears only after qmake is run again by hand. In a git worktree there is no commit id at all — the test is exists(".git/config"), which a worktree fails, so --version prints 3.12.4dev-nogit while git describe --always --dirty in the same tree says 75379ef5-dirty. What attributes a run is git describe --always --dirty plus git status --short, recorded next to the binary; if --version stays in the report, the hint is "run qmake again before the build under test".

macOS — the issue's opening motivation is not in this draft, and it is now measured. A plain qmake && make bundle is unsigned and not sandboxed (codesign -d says code object is not signed at all); a secret file in /tmp works. The same bundle copy, ad-hoc-signed with the in-tree mac/Jamulus.entitlements, exits with status 1 and JSON-RPC: Unable to open secret file /tmp/sbx/secret.txt. Exiting. — the message does not say why. The same file under ~/Library/Containers/app.jamulussoftware.Jamulus/Data/ starts the server. Hint shape: codesign -d --entitlements :- Jamulus.app | grep app-sandbox tells you which case you are in before the first run.

§3 — a probe with no dependencies. Verified against a headless server started as in the draft:

exec 3<>/dev/tcp/127.0.0.1/22998
printf '{"id":1,"jsonrpc":"2.0","method":"jamulus/apiAuth","params":{"secret":"%s"}}\n{"id":2,"jsonrpc":"2.0","method":"jamulus/getVersion","params":{}}\n' "$(cat secret.txt)" >&3
timeout 2 cat <&3

{"id":1,"jsonrpc":"2.0","result":"ok"}
{"id":2,"jsonrpc":"2.0","result":{"version":"3.12.4dev-nogit"}}

A method absent from the build answers {"error":{"code":-32601,"message":"Method not found"}} — the "methods might not exist" case is detectable, not silent.

The client side. The draft's only recipe is a server, and every audio hunk needs a client. One runs on a box with no display and no sound hardware: jackd -d dummy & then ./Jamulus -n -c 127.0.0.1. A CONFIG+=headless serveronly binary cannot be that client — it exits with Client only option(s) '--connect' used.

Editorial: §4 and §7 both say the adversarial input must match the diff; one section would do.