AI worries

[Nitnelav]

Hi there !

It seems like this project is now heavily vibe-coded and it comes with a bunch or quality concerns.

How is the code tested ? I'm sure CI tests pass but did you write them ?
How much code reviewing are you investing in this ?

How do you address those concerns ?

Cheers !

[jmrplens]

Hi, thanks for asking it directly instead of assuming.

Short answer while I write the long one: yes, I use AI for part of the work, and
no, that does not mean anything ships unchecked. Your question is a fair one and
other people will have it too, so it deserves more than a paragraph thrown back
in five minutes. I am writing out how the code is validated, against which
published reference data, and what has to pass before anything merges, with the
links so you can check it rather than take my word for it.

Give me a little while and I will post it here.

[jmrplens]

As promised, here it is in full, with links throughout so you can check any of
it instead of taking my word for it.

Some context first: this project was maintained by me alone for several years
with no AI involved. What changed recently is the pace, not the ownership. I now
use AI for the bulk of the implementation, for refactors and for part of the
documentation. What the library computes, which standard it follows and how a
result is validated are decisions I take and check myself, and every change goes
through review before it lands.

How the code is tested

Unit tests are the smaller half of the answer. There are around 8,000 of them
across 274 test files,
and they are useful, but a passing unit test can easily mean nothing: it can be
written to agree with the implementation instead of with the standard.

The weight is on the conformance suite. Every implemented method is checked
against real reference values: normative tables and worked examples from the
standard itself, and published third-party validation sets where one exists
(more on those below). Each case documents its source in the check. Today that is
533 checks across 57 domains and 362 standards, and the result is a
generated report:

• the report: docs/CONFORMANCE.md
  (also on the site: https://jmrplens.github.io/phonometry/reference/conformance/)
• the checks themselves: scripts/conformance_report.py
• how it is kept honest: CONTRIBUTING, "Conformance report"

If the library drifts from what a third-party source says it should compute, the
conformance job fails. CI regenerates the report on every pull request and fails
if the committed one is stale, so the numbers on the site are always the numbers
the code produced on that commit.

What the validation data actually is

"Published third-party sets" is not a vague gesture. The repository carries
complete, named reference sets, several of them the official material for
verifying an implementation of the standard in question, and the suites that
consume them run in CI:

Reference set	What it validates	In the repository
The electronic attachment of ISO 532-1:2017: the standard's own Annex B validation programme and test signals	Zwicker loudness	data · tests
The certified STIPA verification bench from stipa.info (archived page): the signals IEC 60268-16:2020 Annexes A and C prescribe for verifying a STIPA implementation	Speech transmission index	data · tests
The EBU loudness test set (v04), authentic-programme cases	EBU Tech 3341/3342 loudness gating and loudness range	data · tests
The NORAH2 V2.0.74 public release, the EASA rotorcraft-noise research database and prototype outputs	The ECAC Doc 32 rotorcraft chain, end to end against the published single-event cases	data · tests
The European Commission's CNOSSOS-EU road and rail emission test workbooks (CIRCABC), plus the Official Journal coefficient tables	The CNOSSOS-EU emission equations; the Directive itself prints no worked example, so this is their only oracle	data · road tests · rail tests

Most of that material is copyrighted, so I am precise about what is committed.
Each folder has a README giving the provenance file by file, the source, the
terms, and a removal policy. Where the terms allow it, the committed oracle is a
byte-identical extract: 27 of the 49 STIPA bench signals, stored losslessly and
digest-checked against the published samples on every load, and 44 of the ~360
NORAH2 files. Where they do not, it is a derived series: 36 kB of block-loudness
envelopes standing in for 289 MB of EBU audio that may not be redistributed. And
where the source is numbers, a transcription: the ISO 532-1 expected values and
tolerances, and the Official Journal tables. The committed CNOSSOS cases (60 of
the 4,875 road cases, 123 of the 19,872 rail cases) are a subset of workbooks
the model reproduces in full, all 4,875 road rows to 0.005 dB and every rail row
of the twenty usable vehicles to 0.0055 dB, and the subset covers every level of
every factor the workbooks vary.

The committed extract is what CI asserts against, and those assertions never
skip; the only skips are tests that exist solely to exercise a full set and have
no committed counterpart, and they say so. Drop the full original set into
tests/data-local/ and the same suites run against all of it: the whole
49-signal STIPA bench, the original EBU programme audio end to end, the complete
NORAH2 release. pytest prints which copy each dataset resolved to at the top
of every run
(tests/oracle_data.py),
so a green run is never ambiguous about which oracle produced it. The policy is
written down in
CONTRIBUTING, "Oracle data"
and tests/data/README.md.

What else has to pass before anything merges

The CI workflow
is not only the test job. Every pull request also has to survive:

• ruff, mypy over src and scripts, and bandit
• the conformance report being regenerated and byte-compared
• the API reference being regenerated from the docstrings and compared, so the
  documentation cannot drift from the code
  (scripts/generate_api_docs.py)
• every code snippet in the guides being executed, in both languages, over
  3,600 blocks
  (scripts/check_doc_snippets.py)
• every one of the 1,692 committed figures being regenerated with the library
  and compared
  (scripts/check_figures.py)
• the 67 example report fiches being regenerated and compared
  (scripts/check_reports.py)
• the site building, with HTML validation, a link validator, a KaTeX check and
  a WCAG2AA accessibility audit
  (docs workflow)
• the SonarCloud quality gate
  (https://sonarcloud.io/summary/overall?id=jmrplens_phonometry)

The documentation is not decorative here: the guides under
docs/ and on the site
run as code in CI, and every figure in them is produced by the library.

How much review

Every pull request gets an adversarial review pass, plus the automated reviewers
the repository has enabled, and I answer each comment on the record: either the
finding is real and gets fixed in a follow-up commit, or I say why it is a false
positive with the evidence. Nothing merges with an open comment or a red gate.

The errata registry is a by-product of that same process. When a conformance
check does not reproduce a worked example or a tabulated value, the discrepancy
has to be resolved before the code can be trusted: either my implementation is
wrong, or the source is. Most of the time it is mine. When it is not, I go back
to the standard, read the clause and the printed value myself, and only file it
once I can show the arithmetic that the page cannot satisfy.
docs/ERRATA.md
(https://jmrplens.github.io/phonometry/reference/errata/) is the record of
that second case: 81 defects found in published standards, guidance documents,
textbooks and papers, each with the clause, the printed value, the value the
method actually yields, and the evidence behind the claim. Every entry is held
to a higher bar than the rest of the documentation, because it is a permanent
public statement about a named issuing body or a living author.

How you can check any of this yourself

Pick any number the library produces, find it in the conformance report, and
compare it against the clause the row cites. If a value is wrong, that is a bug
report I want: open an issue with the standard, the clause and the expected
number, and it becomes a conformance check. That is the same standard I hold the
code to, whoever or whatever typed it.

---

One practical note while you are clicking through: the package is in the middle
of a module reorganisation, so a few of the file links above may move under me
over the next weeks. If one of them 404s, searching the repository for the file
name finds it in a second, and nothing in the argument depends on where a file
happens to live today. The report, the conformance script and the data folders
keep their paths.

[Nitnelav]

Ok thanks for your repsonese. Maybe therer is not enough human in the loop for me to fully trust the process. And maybe that's a trust issue on my part :)
I'll wait to see the project adoption to get some third party validation I think. In the meantime I can only encourage you to keep working on this is a very ambitious project. Thanks !

[jmrplens]

Thanks for coming back, and for how you put it. Waiting for third-party
validation before you trust a library is the correct instinct, not a character
flaw, and I would rather see that said out loud than assumed quietly.

One thing worth adding.

I do have my own data. Several years in research left me with measurements and
solved cases across a fair number of the methods and standards implemented here,
some of that work published. One concrete example: the impedance-tube and
transmission methods here are ones I first implemented in
A|Lab, the measurement tool I wrote for an
ESA-funded project
on metamaterials for launch noise, validated experimentally at the time.

None of that is in the conformance suite, and that is deliberate.

Validating my own code against my own measurements is not validation, it is a
closed loop. I control the data, I control the implementation, and a passing
check would prove only that I am consistent with myself. It would also turn the
conformance report into a display case for my own work, which is not what a
reader needs from it. So the suite is built on material published by other
people: normative tables and worked examples from the standards themselves, and
third-party validation sets where they exist, with each check naming its source
so anyone can go and disagree with it. Where nobody has published a reference
case at all, the check is anchored to a closed form and the entry names that
anchor, rather than quietly filling the gap with measurements of mine.

On adoption, since that is what you said you would wait for: it has already
happened to A|Lab, if not yet to this library. A group at Seoul National
University of Science and Technology used it for the impedance-tube
measurements in
Han et al., J. Appl. Phys. 138, 224903 (2025),
and cite it as such, with no involvement from me beyond having published the
tool. I am not going to
dress that up as validation of phonometry, because it is not: different
codebase, different scope, and their paper validates their metastructure, not my
software. But it is the shape of the thing you are asking for.

Which brings me to the open invitation, and this goes for anyone reading the
thread, not only you. Outside validation is what this needs and by definition it
is not something I can produce on my own. If you have work with results you
already trust, measurements you have published, a case you solved with other
software, a dataset from a campaign, running it through this library is the
single most useful thing anyone could do with it right now. If the numbers agree,
that is a data point neither of us controls. If they do not, I want to know, and
I would much rather hear it from a real case than from another test I wrote
myself.

The same goes for anything else: a question, a correction, a method you think is
implemented wrong, a standard you work with that is missing. Open an issue or a
discussion and I will take it seriously, including disagreement about how any of
this is built.

Thanks again for the push, and for the encouragement.