Skip to content

Commit 6796da4

Browse files
fix(semantic): a replayed repair needs corroboration in the frame it lands on
A repair learned on one dataset was replayed onto another whose column meant something different, and applied automatically: A: segment = M/F -> learns M -> male B: segment = S/M/L -> becomes S/male/L/male (status=automatic, human_review=False, confidence 0.95) Clothing sizes silently rewritten as genders. The conflict machinery already existed and worked: _conflict_proposal turns a deterministic-vs-memory clash into a high-risk unsafe_ambiguous record that is never auto-applied. But it only fires on disagreement. On B the CategorySynonymExpert correctly abstains -- the column is not gender-like -- and an abstention is not a disagreement, so _merge_proposals passed the replayed repair through carrying the full confidence it earned elsewhere: det_list = det_by_key.get(key) if not det_list: merged.append(mem_p) # no deterministic proposal at all continue Every other branch was careful. Only total silence had no handling, and silence is exactly what happens when the evidence behind a learned repair is absent. The stored column_signature/value_signature would not have caught it: both columns profile as role="categorical", semantic_type=None, free_text=False. A replayed repair whose issue type is context-dependent -- category_synonym, boolean_synonym, reference_value, where the same token means different things in different columns -- is now demoted to a review-required suggestion when nothing in this frame corroborates it. Context-free repairs (encoding_repair, format_alignment, numeric_format and the rest) are untouched: they mean the same thing in every column, so demoting them would cost the feature without buying any safety. sizes + gender memory S/M/L kept, suggested / risk=high / human_review genders + gender memory still auto-applied -- the feature still works Provenance survives the demotion, which matters more for a held decision than an applied one: memory_influenced, model_id ending :memory, backend="memory", evidence kinds memory_replay and a new memory_uncorroborated note carrying the reason. 4 of the 5 new tests fail on main; the one that passes is the feature-still-works control. This also flips five tests that #481 pinned as current behaviour when it found the defect. They were tripwires for the fix and the fix tripped them; each now asserts the repaired behaviour with its history kept in the docstring. Two findings in that file stay pinned because they remain true: freshdata_version is written and never read, and the SQLite store overwrites on a dataset_id collision. Full suite 7098 passed / 0 failed, coverage 94.94%; ruff clean repo-wide.
1 parent b363c39 commit 6796da4

3 files changed

Lines changed: 224 additions & 36 deletions

File tree

‎src/freshdata/semantic/apply.py‎

Lines changed: 59 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@
88

99
from __future__ import annotations
1010

11+
from dataclasses import replace
12+
1113
import pandas as pd
1214

1315
from .._numeric import safe_to_numeric
@@ -69,6 +71,47 @@ def _describe(decision: SemanticPolicyDecision) -> str:
6971
return f"Skipped semantic repair {pair}"
7072

7173

74+
#: Issue types whose correct repair depends on what the *column* means, not on
75+
#: the token alone. ``"M"`` expands to ``"male"`` in a gender column and to
76+
#: ``"medium"`` in a size column; ``"1"`` is True in a flag column and the
77+
#: number one in a count. A repair of one of these kinds, learned on one
78+
#: dataset, is not evidence about a different dataset's column.
79+
_CONTEXT_DEPENDENT_ISSUES = frozenset(
80+
{"category_synonym", "boolean_synonym", "reference_value"}
81+
)
82+
83+
84+
def _uncorroborated_memory_proposal(mem: SemanticProposal, memory: object) -> SemanticProposal:
85+
"""Demote a replayed context-dependent repair that nothing in this frame supports.
86+
87+
The deterministic expert did not merely disagree -- it produced no proposal
88+
at all for this value, because the evidence that justified the repair on the
89+
learning dataset is absent here. An abstention is not corroboration, so the
90+
replayed repair keeps its provenance but is routed to a human instead of
91+
being applied on authority carried over from another frame.
92+
"""
93+
dataset_id = getattr(memory, "dataset_id", "?")
94+
note = SemanticEvidence(
95+
"memory_uncorroborated",
96+
(
97+
f"no deterministic expert proposed {mem.raw_value!r} -> "
98+
f"{mem.proposed_value!r} for column {mem.column!r} in this frame"
99+
),
100+
0.0,
101+
)
102+
return replace(
103+
mem,
104+
risk="high",
105+
human_review=True,
106+
evidence=(*mem.evidence, note),
107+
rationale=(
108+
f"cleaning memory {dataset_id!r} learned {mem.raw_value!r} -> "
109+
f"{mem.proposed_value!r}, but nothing in this frame corroborates it "
110+
f"for column {mem.column!r}; held for review"
111+
),
112+
)
113+
114+
72115
def _conflict_proposal(
73116
det: SemanticProposal, mem: SemanticProposal, memory: object
74117
) -> SemanticProposal:
@@ -106,7 +149,11 @@ def _merge_proposals(
106149
) -> list[SemanticProposal]:
107150
"""Merge deterministic + memory-retrieved proposals, deduping same-key repairs.
108151
109-
Non-colliding proposals from either source pass through unchanged. When both
152+
A memory proposal with no deterministic counterpart is demoted to a
153+
review-required suggestion when its issue type is context-dependent (see
154+
:data:`_CONTEXT_DEPENDENT_ISSUES`): an abstention by every expert means the
155+
evidence that justified the repair is absent from this frame. Other
156+
non-colliding proposals pass through unchanged. When both
110157
a deterministic expert and memory propose the same ``(column, raw_value)``:
111158
if they agree on the proposed value, keep whichever has the higher
112159
confidence (memory wins ties); if they disagree, replace both with one
@@ -128,7 +175,17 @@ def _merge_proposals(
128175
key = (mem_p.column, mem_p.raw_value)
129176
det_list = det_by_key.get(key)
130177
if not det_list:
131-
merged.append(mem_p)
178+
# Nothing in this frame proposed the same repair. For a
179+
# context-dependent issue type that silence is meaningful: the
180+
# evidence behind the learned repair is gone, so replaying it would
181+
# rewrite a column whose meaning may have changed.
182+
if (
183+
mem_p.proposed_value is not None
184+
and mem_p.issue_type in _CONTEXT_DEPENDENT_ISSUES
185+
):
186+
merged.append(_uncorroborated_memory_proposal(mem_p, memory))
187+
else:
188+
merged.append(mem_p)
132189
continue
133190
touched.add(key)
134191
# A flag (proposed_value=None) is an abstention — "this value looks

‎tests/test_memory_conflicts.py‎

Lines changed: 56 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -71,11 +71,12 @@ def semantic_actions(report) -> list:
7171
# --------------------------------------------------------------------------- #
7272

7373

74-
def test_one_token_categorical_replay_rewrites_a_column_that_changed_meaning(
74+
def test_one_token_categorical_replay_does_not_rewrite_a_column_that_changed_meaning(
7575
gender_memory: CleaningMemory,
7676
) -> None:
77-
"""FINDING (S2): memory replay auto-applies ``M -> male`` onto a column of
78-
clothing sizes.
77+
"""A learned ``M -> male`` must not be replayed onto clothing sizes.
78+
79+
This test was written to pin the defect: replay auto-applied the repair.
7980
8081
Nothing in the retrieval path re-validates *why* the repair was learned.
8182
``CategorySynonymExpert`` only proposed ``M -> male`` on dataset A because
@@ -87,18 +88,26 @@ def test_one_token_categorical_replay_rewrites_a_column_that_changed_meaning(
8788
8889
The stored ``column_signature``/``value_signature`` on the memory record
8990
would not have helped either: both datasets profile as
90-
``role="categorical", semantic_type=None, free_text=False``. What is missing
91-
is a check that the *evidence* behind the learned repair still holds.
91+
``role="categorical", semantic_type=None, free_text=False``.
92+
93+
The fix supplies the missing check: a context-dependent repair replayed with
94+
no deterministic corroboration in *this* frame is demoted to a
95+
review-required suggestion, so memory stays evidence rather than authority.
9296
"""
9397
out, report = fd.clean(sizes_frame(), semantic_mode="auto", memory=gender_memory, **CLEAN)
9498

95-
assert out["segment"].tolist() == ["S", "male", "L", "male"] * 3
96-
applied = [a for a in semantic_actions(report) if a.status == "automatic"]
97-
assert len(applied) == 1
98-
action = applied[0]
99-
assert action.metadata["raw_value"] == "M"
100-
assert action.metadata["proposed_value"] == "male"
101-
assert action.human_review is False
99+
assert out["segment"].tolist() == sizes_frame()["segment"].tolist(), (
100+
"clothing sizes must survive a gender memory untouched"
101+
)
102+
assert not [a for a in semantic_actions(report) if a.status == "automatic"]
103+
held = [
104+
a for a in semantic_actions(report)
105+
if a.metadata.get("raw_value") == "M" and a.metadata.get("proposed_value") == "male"
106+
]
107+
assert len(held) == 1
108+
assert held[0].status == "suggested"
109+
assert held[0].risk == "high"
110+
assert held[0].human_review is True
102111

103112
# Control: without the memory, the deterministic layer leaves B alone.
104113
plain, plain_report = fd.clean(sizes_frame(), semantic_mode="auto", **CLEAN)
@@ -109,14 +118,16 @@ def test_one_token_categorical_replay_rewrites_a_column_that_changed_meaning(
109118
def test_normalized_match_also_catches_the_lowercase_spelling(
110119
gender_memory: CleaningMemory,
111120
) -> None:
112-
"""FINDING (S2, same root cause): retrieval matches on the *normalized*
113-
value, so a learned ``"M"`` also rewrites a dominant lowercase ``"m"`` —
114-
and it does so while the deterministic expert is proposing the opposite
115-
direction (``"M" -> "m"``, aligning to the dominant spelling)."""
121+
"""Retrieval matches on the *normalized* value, so a learned ``"M"`` also
122+
reaches a dominant lowercase ``"m"``. That replay is context-dependent and
123+
uncorroborated here, so it is held for review rather than applied, while the
124+
genuinely *conflicting* value still takes the ``unsafe_ambiguous`` path."""
116125
df = frame(["m"] * 8 + ["M"] * 2 + ["S"] * 3 + ["L"] * 3)
117126
out, report = fd.clean(df, semantic_mode="auto", memory=gender_memory, **CLEAN)
118127

119-
assert out["segment"].tolist()[:8] == ["male"] * 8 # the dominant spelling, rewritten
128+
assert out["segment"].tolist()[:8] != ["male"] * 8, (
129+
"an uncorroborated gender replay must not rewrite the dominant spelling"
130+
)
120131
# The genuinely *conflicting* value is handled correctly, though:
121132
conflicts = [
122133
a for a in semantic_actions(report)
@@ -179,19 +190,23 @@ def test_allowed_values_hint_blocks_the_collision(gender_memory: CleaningMemory)
179190
assert not [a for a in semantic_actions(report) if a.status == "automatic"]
180191

181192

182-
def test_semantic_type_hint_alone_does_not_block_the_collision(
193+
def test_a_semantic_type_hint_does_not_revive_the_collision(
183194
gender_memory: CleaningMemory,
184195
) -> None:
185-
"""FINDING (S3): declaring a *semantic type* is not enough — only
186-
``allowed_values``, ``mutable=False`` or column-level protection stops the
187-
replay. The stored ``column_signature.semantic_type`` is never compared
188-
against the live one at retrieval time."""
196+
"""Declaring a semantic type must not re-enable the uncorroborated replay.
197+
198+
The stored ``column_signature.semantic_type`` is still never compared
199+
against the live one at retrieval time -- that remains a gap -- but it no
200+
longer matters for safety here, because the repair is held for review on
201+
the absence of corroboration rather than on any signature check.
202+
"""
189203
context = {"columns": {"segment": {"semantic_type": "category"}}}
190-
out, _ = fd.clean(
204+
out, report = fd.clean(
191205
sizes_frame(), semantic_mode="auto", memory=gender_memory,
192206
semantic_context=context, **CLEAN,
193207
)
194-
assert out["segment"].tolist() == ["S", "male", "L", "male"] * 3
208+
assert out["segment"].tolist() == sizes_frame()["segment"].tolist()
209+
assert not [a for a in semantic_actions(report) if a.status == "automatic"]
195210

196211

197212
def test_mutable_false_blocks_the_collision_and_audits_the_skip(
@@ -254,20 +269,24 @@ def test_every_memory_derived_decision_is_distinguishable_in_the_audit_trail(
254269
gender_memory: CleaningMemory,
255270
) -> None:
256271
"""All five signals hold on the colliding replay, so a reviewer can tell a
257-
memory-derived mutation from a deterministic one without guessing.
272+
memory-derived decision from a deterministic one without guessing.
258273
259-
Note the status nuance: a *semantic* replay carries the policy-gate status
260-
(``"automatic"``), while ``status="approved"`` is what
261-
:func:`freshdata.memory.annotate_report` stamps on replayed **non**-semantic
262-
decisions plus its own ``step="memory"`` summary action.
274+
The provenance must survive the demotion: a replay that is held for review
275+
rather than applied is exactly the case a reviewer has to understand, so
276+
losing the memory markers there would be worse than losing them on an
277+
auto-applied one.
263278
"""
264279
_, report = fd.clean(sizes_frame(), semantic_mode="auto", memory=gender_memory, **CLEAN)
265-
applied = [a for a in semantic_actions(report) if a.status == "automatic"]
266-
assert len(applied) == 1
267-
action = applied[0]
280+
replayed = [
281+
a for a in semantic_actions(report)
282+
if a.metadata.get("backend") == "memory"
283+
]
284+
assert len(replayed) == 1
285+
action = replayed[0]
268286

269287
assert action.memory_influenced is True # signal 1
270-
assert action.status == "automatic" # signal 2 (see docstring)
288+
assert action.status == "suggested" # signal 2: held, not applied
289+
assert action.human_review is True
271290
assert action.model_id == "semantic:category_synonym:memory" # signal 3
272291
kinds = [e["kind"] for e in action.metadata["evidence"]]
273292
assert "memory_replay" in kinds # signal 4
@@ -390,7 +409,10 @@ def test_version_mismatch_is_recorded_but_never_checked(
390409
{**gender_memory.to_dict(), "freshdata_version": "0.0.1"}
391410
)
392411
out, report = fd.clean(sizes_frame(), semantic_mode="auto", memory=ancient, **CLEAN)
393-
assert out["segment"].tolist() == ["S", "male", "L", "male"] * 3
412+
# The version is still never checked -- that is the finding. The values
413+
# survive only because the replay is uncorroborated here, not because the
414+
# stale version was noticed.
415+
assert out["segment"].tolist() == sizes_frame()["segment"].tolist()
394416
assert not [w for w in report.warnings if "version" in w.lower()]
395417

396418

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
"""Memory is evidence, not authority (#FD2-006).
2+
3+
A repair learned on one dataset was replayed onto another whose column meant
4+
something different, and applied automatically:
5+
6+
A: segment = M/F -> learns M -> male
7+
B: segment = S/M/L -> becomes S/male/L/male
8+
9+
The conflict machinery did exist and did work, but only fired on
10+
*disagreement*: ``_conflict_proposal`` turns a deterministic-vs-memory clash
11+
into a high-risk ``unsafe_ambiguous`` record. On B the ``CategorySynonymExpert``
12+
correctly **abstains** -- the column is not gender-like -- and an abstention is
13+
not a disagreement, so the replayed repair passed through carrying the full
14+
confidence it had earned on a different column.
15+
16+
The stored signatures would not have caught it: both columns profile as
17+
``role="categorical", semantic_type=None, free_text=False``.
18+
"""
19+
20+
from __future__ import annotations
21+
22+
import pandas as pd
23+
import pytest
24+
25+
import freshdata as fd
26+
from freshdata.semantic.apply import _CONTEXT_DEPENDENT_ISSUES
27+
28+
CLEAN = {"verbose": False, "return_report": True}
29+
30+
31+
def _frame(values) -> pd.DataFrame:
32+
return pd.DataFrame(
33+
{"id": [f"r{i}" for i in range(len(values))], "segment": list(values)}
34+
)
35+
36+
37+
@pytest.fixture
38+
def gender_memory():
39+
"""Memory learned where ``segment`` genuinely holds genders."""
40+
learn_df = _frame(["M", "F"] * 6)
41+
_, report = fd.clean(learn_df, semantic_mode="auto", **CLEAN)
42+
return fd.learn_cleaning_memory(learn_df, decisions=report, dataset_id="hr")
43+
44+
45+
def _semantic(report):
46+
return [a for a in report.actions if a.step == "semantic"]
47+
48+
49+
def test_a_gender_memory_does_not_rewrite_clothing_sizes(gender_memory):
50+
"""The headline trap: S/M/L must survive a memory that learned M -> male."""
51+
sizes = _frame(["S", "M", "L", "M"] * 3)
52+
out, report = fd.clean(sizes, semantic_mode="auto", memory=gender_memory, **CLEAN)
53+
54+
assert out["segment"].tolist() == sizes["segment"].tolist()
55+
assert not [a for a in _semantic(report) if a.status == "automatic"]
56+
57+
58+
def test_the_held_repair_is_routed_to_a_human_with_its_reason(gender_memory):
59+
"""Holding it silently would only trade one failure for another."""
60+
sizes = _frame(["S", "M", "L", "M"] * 3)
61+
_, report = fd.clean(sizes, semantic_mode="auto", memory=gender_memory, **CLEAN)
62+
63+
held = [a for a in _semantic(report) if a.metadata.get("backend") == "memory"]
64+
assert len(held) == 1
65+
action = held[0]
66+
assert action.status == "suggested"
67+
assert action.risk == "high"
68+
assert action.human_review is True
69+
assert "corroborate" in action.rationale
70+
assert action.metadata["raw_value"] == "M"
71+
assert action.metadata["proposed_value"] == "male"
72+
73+
74+
def test_replay_onto_the_same_kind_of_column_still_works(gender_memory):
75+
"""The guard must not disable the feature it is guarding.
76+
77+
A second genuinely gender-shaped frame is exactly what memory is for, and
78+
there the deterministic expert corroborates the repair.
79+
"""
80+
genders = _frame(["M", "F", "M", "F"] * 3)
81+
out, report = fd.clean(genders, semantic_mode="auto", memory=gender_memory, **CLEAN)
82+
83+
assert out["segment"].tolist() == ["male", "female"] * 6
84+
assert [a for a in _semantic(report) if a.status == "automatic"]
85+
86+
87+
def test_provenance_survives_the_demotion(gender_memory):
88+
"""A held replay is precisely the case a reviewer must be able to trace."""
89+
sizes = _frame(["S", "M", "L", "M"] * 3)
90+
_, report = fd.clean(sizes, semantic_mode="auto", memory=gender_memory, **CLEAN)
91+
92+
action = next(a for a in _semantic(report) if a.metadata.get("backend") == "memory")
93+
assert action.memory_influenced is True
94+
assert action.model_id.endswith(":memory")
95+
kinds = [e["kind"] for e in action.metadata["evidence"]]
96+
assert "memory_replay" in kinds
97+
assert "memory_uncorroborated" in kinds, "the reason for holding must be recorded"
98+
99+
100+
def test_a_context_free_repair_is_not_demoted(gender_memory):
101+
"""Only context-dependent issue types need corroboration.
102+
103+
An encoding or format repair means the same thing in every column, so
104+
demoting it would make memory useless without making anything safer.
105+
"""
106+
assert "category_synonym" in _CONTEXT_DEPENDENT_ISSUES
107+
assert "boolean_synonym" in _CONTEXT_DEPENDENT_ISSUES
108+
for context_free in ("encoding_repair", "format_alignment", "numeric_format"):
109+
assert context_free not in _CONTEXT_DEPENDENT_ISSUES

0 commit comments

Comments
 (0)