Skip to content

test(corpus): exploration query family (q055-q059) — naive first-touch profiling#1506

Open
aaj3f wants to merge 4 commits into
perf/r2rml-loadtable-meta-cachefrom
test/exploration-corpus-family
Open

test(corpus): exploration query family (q055-q059) — naive first-touch profiling#1506
aaj3f wants to merge 4 commits into
perf/r2rml-loadtable-meta-cachefrom
test/exploration-corpus-family

Conversation

@aaj3f

@aaj3f aaj3f commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

What

Adds the five naive first-touch queries every human or LLM agent runs on an unknown dataset — the BI-derived corpus had zero pure-wildcard coverage. New closed Tag variants Wildcard and Exploration; native oracles blessed; honest per-member status taken from the actual run.

id query native virtual rows gate
q055 ?s ?p ?o LIMIT 5 1 ms 2.44 s 5 rows_only
q056 COUNT(*){?s ?p ?o} 0 ms 168.29 s 1 full
q057 DISTINCT ?p LIMIT 100 0.9 s 34.93 s 100 rows_only
q058 ?s a ?type census GROUP BY 0.15 s 5.40 s 16 full
q059 predicate histogram GROUP BY ?p 2.6 s 32.38 s 10 full

Measured virtual-sf01 vs native-sf01, hot cache, 1 rep, on the #1503 head (6750286df). All five complete ok — none DNF (the predicted DNFs for q056/q058/q059 were overturned by measurement); expected_status stays ok/ok for all five.

Mechanisms

All five hit the same whole-dataset breadth (~69,046 files across all 16 TriplesMaps — the maximum-fan-out first-touch case). The wall splits by what columns get materialized:

  • q056 (168 s) is the headline gap — predicate-less COUNT(*) has no manifest COUNT shortcut (not a single resolved table), so it materializes every triple: full object-column fetch, fetch_bytes 63× that of q057 over the same file set. It reads object values it only needs to count — same family as q038's un-fused COUNT. A fused predicate-less COUNT(*) (Σ over TriplesMaps of rows × POMs from manifest record counts) would take it to sub-second (PR-6 family).
  • q057 / q059 (32–35 s) — only ?p needed → object columns pruned → per-file footer/decode overhead across 69,046 files dominates. This is PR-2a (the per-file decode-wall) at maximum fan-out.
  • q058 (5.4 s)?s a ?type needs only subject-key + per-table rr:class constant → 16 scans (one per TriplesMap), column-pruned. The cheapest wildcard, and why the census DNF prediction was wrong.
  • q055 (2.4 s)LIMIT 5 doesn't prune the scan but stops after one fact table (~7,679 files, 63 MB).

Gate decisions

  • q055, q057 → rows_only. Both truncate an unordered set with LIMIT, so two correct engines return different-but-valid subsets that can't be hash-compared. q057 is empirical — it was the sole parity failure in compare, and the mapping declares ~120 distinct predicates (119 rr:predicate + rdf:type) > the LIMIT 100, so the 100-subset is nondeterministic.
  • q056, q058, q059 → full hash; all three matched native exactly. q059 is deterministic despite the LIMIT (ORDER BY DESC(?n) ?p tiebreak).
  • q056 timeout_s 300 — measured 168.3 s hot; 300 s is CI-stability headroom (168/180 = 93% is fragile), not an expectation. The wall is recorded honestly in the ROADMAP row.

Commits

  1. test(corpus): fix error-boundary drift — declared_error must include q013 — a pre-existing test/manifest drift: the manifest declared q013 virtual=error but the error_boundary self-test's expected set listed only q034/q051. Isolated as its own commit.
  2. test(corpus): add the exploration query family (q055-q059) — the two tags, 5 .rq files, manifest entries, and blessed native oracles.
  3. docs(roadmap): exploration family section — measured walls + mechanisms, kept OUT of the north-star 44/51 accounting pending AJ's ruling on whether wildcard first-touch queries join the ≤3 s bar.

Stacked on #1503 (perf/r2rml-loadtable-meta-cache). Additive-only — no existing blessed oracle or manifest entry changed. Corpus self-tests green (7 passed).

aaj3f added 4 commits July 15, 2026 13:36
…q013

The manifest declares three queries expected_status.virtual=error (q013
subquery, q034 transitive path, q051 subquery), but the error_boundary
self-test's expected set listed only q034/q051, so the assertion no
longer matched the manifest it validates. Add q013 to the expected set
and to the native-stays-ok loop.

Isolated from the exploration-family change per review: this corrects a
pre-existing manifest/test drift and stands on its own.
The BI-derived corpus had zero pure-wildcard coverage, yet an
unconstrained ?s ?p ?o is the first query every human or LLM agent runs
on an unknown dataset. Add the five naive first-touch profiling queries
and two closed Tag variants (Wildcard, Exploration):

  q055 wildcard peek        SELECT ?s ?p ?o LIMIT 5
  q056 triple count         SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }
  q057 schema discovery     SELECT DISTINCT ?p ... LIMIT 100
  q058 class census         SELECT ?type (COUNT(?s) ...) GROUP BY ?type
  q059 predicate histogram  SELECT ?p (COUNT(*) ...) GROUP BY ?p ORDER BY DESC(?n) ?p LIMIT 10

Native oracles blessed; honest per-member status from the actual run
(virtual-sf01 vs native-sf01, hot cache, 1 rep, HEAD 6750286):

  q055    1ms /   2.44s    5 rows
  q056    0ms / 168.29s    1 row   (predicate-less COUNT — full
                                    materialization of every triple)
  q057  900ms /  34.93s  100 rows
  q058  147ms /   5.40s   16 rows
  q059  2.6s  /  32.38s   10 rows

All five complete ok within timeout — none DNF (the earlier DNF
prediction for q056/q058/q059 was overturned by measurement).
expected_status stays ok/ok (omitted) for all five.

Gate decisions:
- q055, q057 -> rows_only. Both truncate an unordered set with LIMIT, so
  two correct engines return different-but-valid subsets that cannot be
  hash-compared. q057 is empirical: it was the sole parity failure in
  compare, and the mapping declares ~120 distinct predicates (119
  rr:predicate + rdf:type) > the LIMIT 100, so the 100-subset is
  nondeterministic.
- q056, q058, q059 -> full hash; all three matched native exactly. q059
  is deterministic despite LIMIT (ORDER BY DESC(?n) ?p tiebreak).
- q056 timeout_s 300 (measured 168.3s hot; 300s is CI-stability headroom
  at the 93%-of-180s fragility, not an expectation).
… north-star

Records the naive first-touch family's measured walls and mechanisms in
its own section, deliberately OUT of the north-star 44/51 accounting
(pending AJ's ruling on whether wildcard first-touch queries join the
≤3s bar). Headline: q056's predicate-less COUNT at 168s is the real
first-touch cost — full object materialization of every triple across
all 16 tables (69,046 files, 63x the fetch_bytes of the object-pruned
q057 over the same file set), a PR-6-family un-fused-COUNT gap, not a
hard failure. All five complete ok; the earlier DNF prediction was
overturned by measurement. Notes q056's 300s gate as CI-stability
headroom (168/180 = 93% fragility) and flags q029's same-ratio gate as
a candidate for the same treatment when F17 lands.
… + q056 F22 register row

- q055/q057: add the '# Determinism:' header line the q029/q045 family carries,
  mirroring the manifest self-test's rows_only rationale (unordered LIMIT
  truncation) into the query files so the convention is uniform.
- 04-findings-register.md: add F22 for q056's un-fused predicate-less COUNT
  (168.3s materializes every triple to count rows it never inspects), pointing at
  the fused Σ(record_count × POM) fix class so it sits beside its q038 sibling.
  Exploration finding — no burndown row (out of the north-star count).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant